Entries tagged php

Automate selling at LaRedoute #2: Parse order files

Posted on 16. Dezember 2014 Comments

This blog post is part of the series Automize selling at LaRedoute.

  • Part 1: Get new orders
  • Part 2: Parse order files
  • Part 3: Upload response files
  • Part 4: update quantity and price feed

Update 2016: La Redoute is going to stop using CSV and moves everything to SOAP Webservices. Tutorials will follow


In part 1 the files containing orders are downloaded into a folder called OrdersFromLaRedoute. The next script is going to go through that folder, parse the file and insert it into a table.

These are the values the table must have because we’re just going to insert everything that’s in the CSV files.


$dbValues = ['MarketplaceID', 'OrderID', 'StorefrontOrderID', 'OrderDate', 'BuyerEmailAddress', 'BuyerName', 'BuyerPhoneNumber', 'OrderItemCode', 'ItemStatus', 'SKU', 'Title', 'Quantity', 'ItemPrice', 'ItemTax', 'ShippingCharge', 'ShippingTax','ItemFee', 'Currency', 'ShippingOption', 'PaymentInfo', 'ShippingAddressName', 'ShippingAddressFieldOne', 'ShippingAddressFieldTwo', 'ShippingAddressFieldThree', 'ShippingCity', 'ShippingStateOrRegion', 'ShippingPostalCode', 'ShippingCountryCode', 'ShippingPhoneNumber', 'BillingAddressName', 'BillingAddressFieldOne', 'BillingAddressFieldTwo', 'BillingAddressFieldThree', 'BillingCity', 'BillingStateOrRegion', 'BillingPostalCode', 'BillingCountryCode', 'BillingPhoneNumber'];

 

Therefore I created the table more or less like this:

CREATE TABLE `ORDERS-HISTORY` (
`MarketplaceID` int(11) DEFAULT NULL,
`OrderID` varchar(50) DEFAULT NULL,
`StorefrontOrderID` varchar(50) DEFAULT NULL,
`OrderDate` varchar(50) DEFAULT NULL,
`BuyerEmailAddress` varchar(50) DEFAULT NULL,
`BuyerName` varchar(50) DEFAULT NULL,
`BuyerPhoneNumber` varchar(50) DEFAULT NULL,
`OrderItemCode` varchar(50) NOT NULL DEFAULT '',
`ItemStatus` varchar(10) NOT NULL DEFAULT '',
`SKU` int(11) DEFAULT NULL,
`Title` varchar(50) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`ItemPrice` decimal(8,2) DEFAULT NULL,
`ItemTax` decimal(8,2) DEFAULT NULL,
`ShippingCharge` decimal(8,2) DEFAULT NULL,
`ShippingTax` decimal(8,2) DEFAULT NULL,
`ItemFee` decimal(8,2) DEFAULT NULL,
`Currency` varchar(3) DEFAULT NULL,
`ShippingOption` varchar(10) DEFAULT NULL,
`PaymentInfo` varchar(50) DEFAULT NULL,
`ShippingAddressName` varchar(70) DEFAULT NULL,
`ShippingAddressFieldOne` varchar(70) DEFAULT NULL,
`ShippingAddressFieldTwo` varchar(70) DEFAULT NULL,
`ShippingAddressFieldThree` varchar(70) DEFAULT NULL,
`ShippingCity` varchar(70) DEFAULT NULL,
`ShippingStateOrRegion` varchar(70) DEFAULT NULL,
`ShippingPostalCode` int(11) DEFAULT NULL,
`ShippingCountryCode` varchar(3) DEFAULT NULL,
`ShippingPhoneNumber` varchar(50) DEFAULT NULL,
`BillingAddressName` varchar(70) DEFAULT NULL,
`BillingAddressFieldOne` varchar(70) DEFAULT NULL,
`BillingAddressFieldTwo` varchar(70) DEFAULT NULL,
`BillingAddressFieldThree` varchar(70) DEFAULT NULL,
`BillingCity` varchar(70) DEFAULT NULL,
`BillingStateOrRegion` varchar(70) DEFAULT NULL,
`BillingPostalCode` int(11) DEFAULT NULL,
`BillingCountryCode` varchar(3) DEFAULT NULL,
`BillingPhoneNumber` varchar(50) DEFAULT NULL,
PRIMARY KEY (`OrderID`,`OrderItemCode`,`ItemStatus`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 

As you can see, the primary key consists of OrderID, OrderItemCode and ItemStatus. An OrderID is the unique identifier of one order, consisting of possibly many but at least one OrderItemCode. An OrderItemCode is representing an SKU + Quantity. For newly created orders, the ItemStatus will appear as „Created“. Once an item is accepted, LaRedoute will put a file into the ToSupplier folder with exactly the same OrderID, OrderItemCode but the ItemStatus „ToShip“. This will be important in Step 3.

Then, use a CSV library like league/csv and insert it with e.g. medoo.


$dirAsArray = scandir("OrdersFromLaRedoute");

foreach($dirAsArray as $file) {
// parse and INSERT
}

Automate selling at LaRedoute #1: Get new orders

Posted on 16. Dezember 2014 Comments

This blog post is part of the series Automize selling at LaRedoute.


The french marketplace LaRedoute unfortunately doesn’t have a real API, but they do have ways to automize some processes. A lot of smaller marketplaces have this concept as well. You will get credentials for an SFTP server. On this server you will find the folders ToSupplier and FromSupplier, where the „supplier“ (aka you) can up- and download a range files documented by Merchantry in their blog. The processing of the uploaded files can take up to 6 hours, but is sometimes done in only a couple of minutes, so I’m going to assume the worst case of 6 hours in this post.

While programming a couple of scripts I found the following problems:

  • the server is incredibly slow sometimes (better at nights), so sometimes the connections just time out
  • sometimes the listing for the ToSupplier folder times out because there are too many files (according to support…huh?), so they have to be deleted regularly
  • not only the connection to LaRedoute but also the connection to my local MySQL server times out
  • I have to reserve a purchased item once I accepted it on LaRedoute immediately, because it could be sold elsewhere in the 6 hours LaRedoute might take to give me the shipping address

New orders can be found in the ToSupplier folder in tab seperated CSV files (but .txt ending) with the format OrdersYYYY-MM-DD-hh-mm-ss.txt.

Since PHP is the companies main language I will show a couple of scripts which automize downloading and processing those files. The code is of course simplified for better understanding. We’re using SFTP instead of FTP and I found using the phpseclib to be the most usable library.

I will propose the use of 2 Tables in the MySQL database: TEMP-FILENAMES and FILENAMES-HISTORY. Both have a the unique column filename. FILENAMES-HISTORY will contain the name of every file ever processed by the following script, TEMP-FILENAMES is a helper table that will be truncated after every run.

First, we need to establish a connection


$sftp = new Net_SFTP(SFTP_LAREDOUTE_HOST);
if (!$sftp->login(SFTP_LAREDOUTE_USER, SFTP_LAREDOUTE_PASS)) {
exit('Login Failed');
}

Then we change directory. This is a command that usually involves listing the directory changed into, but since this is not a graphical client, the real timeout might come on line below. $nlist will just be null if the listing fails, and I will assume it didn’t work if it takes more than 30 seconds.

$sftp->chdir('/ToSupplier');

$beforetime = time();
$nlist = $sftp->nlist();
$aftertime = time();
if(($aftertime-$beforetime) > 30 ) {
exit('Timeout while Listing directory');
}

The next piece of code is only executed if the listing worked. Every filename that includes the word „Order“ is now inserted into the temporary table:

foreach($nlist as $filename) {
if (strpos($filename, 'Order') !== false) {
$qry = "INSERT INTO `TEMP-FILENAMES`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry,MYSQLCONNECTION) or print mysql_error();
}
}

You can look at the difference between the filenames in your HISTORY table and the possibly new ones in the temporary table.

$tmpCmpFilenames = array();
$qry = "SELECT `filename` FROM `TEMP-FILENAMES` WHERE `filename` NOT IN (SELECT `filename` FROM `FILENAMES-HISTORY`)";
$select = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();
while ($row = mysql_fetch_assoc($select)) {
$tmpCmpFilenames[] = $row['filename-id'];
}

Now we have all the new files in the array $tmpCmpFilenames. The correct way would be make sure the downloaded files are correct with hashes. Instead we decided to misuse the filesize, since it’s a good indicator something didn’t work properly;) The files not downloaded correctly are deleted from the array. They will appear next time the script is run.

foreach($tmpCmpFilenames as $filename) {
$remotefilesize = $sftp->size($filename);
$sftp->get($filename, 'OrdersFromLaRedoute/' . $filename);
$localfilesize = filesize('OrdersFromLaRedoute/' . $filename);
if ($remotefilesize != $localfilesize) {
unset($tmpCmpFilenames[$filename]);
}
}

We can now insert the filenames into the HISTORY table.

foreach($tmpCmpFilenames as $filename) {
$qry = "INSERT INTO `FILENAMES-HISTORY`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();
}

Last but not least, the temporary table needs to be truncated for the next run.

$truncate=mysql_query("TRUNCATE TABLE `TEMP-FILENAMES`",MYSQLCONNECTION) or print mysql_error();

The next step is described in part 2 of this series.

Anleitung: Produkte bei eBay über API mit PHP SDK listen – Teil 2: XML Dateien mit PHP erstellen

Posted on 10. Oktober 2014 Comments

Dieser Blog Post ist Teil der Reihe Produkte bei eBay listen.


2.1. XMLWriter

Für das Schreiben von XML in PHP wird hier die XMLWriter Klasse verwendet, sie sollte eigentlich überall vorhanden sein. Als erstes wird ein Objekt erstellt.

$writer = new XMLWriter();

Mit dem folgenden Code kann man zwischen Ausgabe im Browser/auf der Konsole und dem Schreiben in eine .xml-Datei hin- und herschalten.

if ($DEBUG) {
$writer->openURI('php://output');
} else {
$filename = 'AddFixedPriceItem.xml';
touch($filename);
$writer->openURI($filename);
}

Der XMLWriter macht allerdings keine Absätze und so würden die folgende Anweisungen alles in eine Zeile schreiben. Prinzipiell ist das natürlich erstmal nicht unbedingt ein Problem. Allerdings wird eBay die Datei ab einer bestimmten Zeilenlänge nicht mehr akzeptieren, (wahrscheinlich) da die Zeilenanzahl ein Kriterium für die maximale Größe von BulkDataExchangeRequests sind.

$writer->setIndent(true);

Bevor die Elemente geschrieben werden, wird erst einmal das Dokument mit Version und Encoding begonnen.

$writer->startDocument('1.0', 'UTF-8');

Ergebnis:

<?xml version="1.0" encoding="UTF-8"?>

Ab jetzt können beliebig Elemente geschrieben werden. Übergeordnete Elemente können mit den folgenden Befehlen geöffnet und geschlossen werden. Dabei ist beim Schließen der Name egal, es zählt die Reihenfolge. Gerade in Schleifen sollte man hier also genau hinsehen.

$writer->startElement('BulkDataExchangeRequests');
...
$writer->endElement();

Ergebnis:

<BulkDataExchangeRequest>
...
</BulkDataExchangeRequest>

Soll ein Element mit einem Wert geschrieben werden wird der folgende Befehl verwendet

$writer->writeElement('SiteID', '77');

Ergebnis

<SiteID>77</SiteID>

Nun gibt es noch den seltenen Fall, dass in einem Tag noch ein Attribut vorhanden ist. Dies wird folgendermaßen realisiert:

$writer->startElement('ShippingServiceCost');
$writer->writeAttribute('currency', 'EUR');
$writer->text('0.0');
$writer->endElement();

Ergebnis:

<ShippingServiceCost currency="EUR">0.0</ShippingServiceCost>

Zuletzt sollte das Dokument noch geschlossen und der Puffer geschrieben (entweder in die Ausgabe oder in die Datei) werden:

$writer->endDocument();
$writer->flush();

2.2. Dateien zippen

In den Beispielen, die in Teil 3 benutzt werden, wird die .xml-Datei noch komprimiert, bevor sie hochgeladen wird. Dies kann mit dem folgenden Snippet umgesetzt werden.

if (!$DEBUG) {
$gzfile = $filename . ".gz";
$fp = gzopen($gzfile, 'w9');
gzwrite($fp, file_get_contents($filename));
gzclose($fp);
}

Updating Owncloud News App with cron and Uberspace

Posted on 27. November 2013 Comments

Since it’s not possible to create a cronjob with the user the webserver is running under(as advised), I just created a cronjob with my user(crontab -e) and it works just fine.

[repat@uberspace]$ crontab -l
*/15 * * * * /package/host/localhost/php/bin/php -f /home/repat/html/owncloud/cron.php

The problem was the following: owncloud wouldn’t update my news and I didn’t think it was a cron problem, since I couldn’t update it manually(AJAX). Adding new feeds however did work. I still don’t get why manual updating doesn’t work , even with the Android app. Opening cron.php in the webbrowser didn’t work, although it did display {„status“:“success“}..turns out: it has to be called via CLI.

Unfortunately, I get an email every 15 minutes now… To disable that I added this above the cron entry:

MAILTO=""

Thanks to Raydiation and j-ed for helping me.