Entries filed under English

Quick and dirty redirect script for the fitting shipping provider

Posted on 2. März 2017 Comments

Another way to use the PHP library shipping-service-provider-check apart from fixing the shipping providers in the ERP Plentymarkets is a simple quick and dirty PHP script that redirects to the tracking page of a fitting shipping provider. You can find the whole sourcecode on GitHub.

Installation

If you don’t use some sort of MVC this is a quick way to do it. If you do, I suspect you’ll know how to alter this example to fit your system. Feel free to write me an e-mail or comment if you need help

  1. Log onto your server and create a folder
  2. Copy or git pull the files, including .htaccess
  3. Get Composer
  4. composer require repat/shipping-service-provider-check

Explanation of the code

The $trackingId variable is received via HTTP GET. This means the URL would be http://url.tld/trackingscriptfolder/script.php?tracking_id=123456 (or without the script.php in case of the .htaccess), where 123456 is the TrackingID. The rest is more or less just for debugging. The scripts returns a HTTP Code 422 and ends if there is no input given.

$trackingId = $_GET["tracking_id"];
if (empty($trackingId)) {
  $unprocessableEntity = 422;
  http_response_code($unprocessableEntity);
  echo "wrong input";
  return;
}

Next the URLs will be defined. The keys have to match the shipping providers so make sure the shipping providers are added at the library as well. The TrackingID will be added at the end of the URL, so make sure they have the right format.

$shippingProviderURLs = [
  "dhl" => 'https://nolp.dhl.de/nextt-online-public/set_identcodes.do?idc=',
...
];

Then follows the documented way of checking for the providers. It could technically be that at the same time a TrackingID is valid for more than one provider. To keep things simple, we’ll just use the first one (array_search instead of array_keys). In case there is none found, array_search returns false and the script continues. In case there is one found, the header() function is called to redirect the user to the correct provider page.

$checkedProvider = array_search(true, $result);
$urlOfCheckedProvider = $shippingProviderURLs[$checkedProvider];

if ($checkedProvider !== false) {
  header('location: ' . $urlOfCheckedProvider . $trackingId);
}

Now it’s up to you what you will do with the ones you can’t redirect. I decided to write a little skeleton.html file and include that with a simple message that the user could try the shipping providers where a check is not possible (no API, no website scraping, no regex). They are listed earlier:

$shippingProviderURLsNoCheck = [
  "DPD" => 'https://tracking.dpd.de/parcelstatus?query=',
...

Flattr this!

Fix shipping providers for FBA orders in Plentymarkets

Posted on 27. Februar 2017 Comments

The popular ERP Plentymarkes let’s you send your orders via the Fulfillment by Amazon program to your customers. Amazon is just used as a logistics partner, the order doesn’t have to come via Amazon. It could e.g. be a Rakuten or eBay order. The problem that Plentymarkets didn’t address at all and is discussed in the private forums, is that Amazon chooses whatever shipping company they seem fit.

Every order has a standard shipping provider and with it comes a URL for tracking the package. Most of the vendors would implement an event procedure („Ereignisaktion“) that sends an email to the customer containing the order and the TrackingID as soon as it’s shipped. The TrackingIDs are directly imported from Amazon. Unfortunately Plentymarkets does not change the shipping provider in case it’s not the default one (which is often the case). So the customer gets wrong information.

In one of last last blogposts I introduced a PHP package to check which shipping provider a TrackingID belongs to by either scraping their website or using a regular expression and filtering for common patterns. In this blog post I want to explain how to use it to fix those orders. You will need a webserver with PHP and MySQL and a little knowledge of programming and Linux.

You can find some simplified code examples on GitHub soon.

  1. Take your FBA orders out of the event procedures that sends the email to the customers, filter e.g. via WarehouseID
  2. Instead, use an event procedure to mark those orders with a flag, e.g. a star (ID 2)
  3. Create a DynamicExport of the type OrderComplete. Filter for your Sent Status (usually 7) and WarehouseID of FBA orders. Include these fields:
    1. OrderID
    2. OrderPackageNo
    3. OrderParcelServiceID
    4. OrderParcelServicePresetID
    5. OrderLastUpdateTimestamp
  4. Implement the SOAP Call GetDynamicExport (will be replaced by REST via Plugin mid-2017)
  5. Write a Cronjob that calls the GetDynamicExport script with the FormatID of the DynamicExport from #2
  6. Create a database table called ‚pm_shipping_providers_check‚ with the fields from #2
  7. Write a script that imports the CSV you downloaded with the GetDynamicExport call into your MySQL database
  8. Create another DynamicExport, this time of the type Order. Include these fields:
    1. OrderID (Synchronisation)
    2. ParcelServiceID (Import)
    3. ParcelServicePresetID (Import)
    4. PackageNo (Import)
  9. Create a table with these fields called ‚pm_shipping_providers_correction
  10. Create a script, that uses the shipping-service-provider-check library to check the every PackageNo in ‚pm_shipping_providers_check‘ and – if the provider is different from the default one – writes it into ‚pm_shipping_providers_correction‘
  11. Implement the SOAP Call SetDynamicImport (see #3)
  12. Write a script that exports the table ‚pm_shipping_providers_correction‘ into a (semicolon seperated) CSV.
  13. Create a cronjob that uploads the file regularly
  14. Create an event procedure on the event PackageNo that sends out emails, but only for the previously flagged orders

Flattr this!

Simple wget crawler for list of files

Posted on 6. August 2015 Comments

This script could be helpful to download a set of files from a webserver, that you don’t have (S)FTP access to. The input file consists of a list of filenames, one name each line.


#!/bin/bash
file=$1
WEBSERVER="http://webserver.tld/folder/"
while IFS= read -r line; do
FULLURL="$WEBSERVER$line"
wget -nc -R --spider $FULLURL
done < "$file"

At first, the first command line argument is saved into the variable file. Then the Webserver address is saved to the WEBSERVER variable. IFS stands for Internal Field Separator. It’s used to read line by line through the file in the while loop that ends in the last line. Inside of the loop, the read line is concatenated with the webserver address into FULLURL. Then, wget is used with the parameters -nc for checking if the file is not already present in the current folder, -R for downloading and –spider for checking the existence on the webserver.

You can find the script on GitHub.

Flattr this!

Collect currency exchange rates in a MySQL database with PHP and fixer.io API

Posted on 4. August 2015 Comments

If you work in a company that buys and sells goods in many different currencies, it might be a good idea, to use the latest exchange rates. Also, it might be useful, to store old exchange rates to clarify/verify old business decisions. If once a day is enough for you, fixer.io offers a free simple Rest API. A lot of the code at my work is written in PHP but I usually use the request library in JavaScript and Python, so I’m using it in this example too. A common PHP solution would be guzzle. But first, get composer (the PHP counterpart to npm or pip):

$ curl -sS https://getcomposer.org/installer | php

$ php composer.phar require rmccue/requests

The mysql_ commands are deprecated (and removed in PHP 7), use mysqli or PDO. Also you should use some sort of framework for the database access, like medoo or a proper ORM. This is just proof of concept.


$base = 'EUR';
$request = Requests::get('https://api.fixer.io/latest?base=' . $base, array('Accept' => 'application/json'));
if ($request->status_code == 200) {
$response = json_decode($request->body);
$GBP = $response->rates->GBP;
$CAD = $response->rates->CAD;
$USD = $response->rates->USD;
$NOK = $response->rates->NOK;
$CNY = $response->rates->CNY;
$rBase = mysql_real_escape_string($response->base);
$date = mysql_real_escape_string($response->date);
$currencies = mysql_real_escape_string("1.0, $USD, $GBP, $NOK, $CNY, $CAD");
$qry = "INSERT INTO `exchange_rates_fixerio`(date, base, eur, usd, gbp, nok, cny, cad) VALUES ('$date', '$rBase', $currencies);";
$insert = mysql_query($qry, $mysqlConnection) or print mysql_error();
}

I assume the database connection is defined earlier, there’s lot’s of documentation for that. Because we are from Europe, I chose Euro (EUR) as the base currency. Apart from the get() method, you need nothing else, to send a request. If the request returns an OK(200), the response is read and saved into different variables, e.g. for British Pounds, US Dollar, Canadian Dollar, Chinese Renminbi and Norwegian Krone. Just to make sure we have the right base, it’s also parsed. From there it’s only a simple INSERT INTO (as said before, use a framework for that)

The table could look like this:

CREATE TABLE `echange_rates_fixerio` (
`date` date NOT NULL,
`base` varchar(3) NOT NULL,
`eur` double NOT NULL,
`usd` double NOT NULL,
`gbp` double NOT NULL,
`nok` double NOT NULL,
`cny` double NOT NULL,
`cad` double NOT NULL
)

 

You can also find this code on GitHub.

Flattr this!

Using nanoc for podcast feeds

Posted on 23. Juli 2015 Comments

Nanoc is a static site generator much like jekyll or octopress, but with a more minimalistic approach. These generators are not necessarily the most suitable choice for a podcast website, but it’s possible and you might save up on webspace and traffic when you use GitHub or Neocities.

Creating the podcast feed is basically like writing a normal Atom feed for the blog, since podcast feeds ARE indeed feeds with an enclosure tag in which the URL to the audio or video file is placed. This guide does not include the itunes tags. I might add it one day. Follow the instructions and use the documentation for the Helper Blogging. Tag your podcast episodes as kind:article.

The media files are placed in content/mp3 and content/opus, which is where the links in the feeds will point to later.

I invented the fields mp3 and opus, since these are the file formats I want to use. The values are the filenames. The header of a new episode/post would look like

---
title: 001 - Podcast Episode Title
created_at: 2015-03-14 09:00:00 +0000
kind: article
tags: [podcast,topic]
mp3: 001-podcast-episode-title.mp3
opus: 001-podcast-episode-title.opus
---

This has to be filled manually everytime, so make sure you have the exact filename, as some podcast clients won’t allow correction of the URL.

The next step is to write different feeds for the formats. For that, I’m using the new field format, which will be interpreted by the Helper class later on. For example, I called my normal feed blogfeed and the podcast feeds mp3feed and opusfeed. Create the file blogfeed.erb in the content folder and fill it with the following:

<%= atom_feed :title => 'repats podcast blog', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'blog' %>

The mp3feed.erb and opusfeed.erb are filled accordingly:

<%= atom_feed :title => 'repats podcast mp3', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'mp3' %>

<%= atom_feed :title => 'repats podcast opus', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'opus' %>

The next step is to use the Blogging locally in your nanoc installation. To do that you need to copy it from the gems folder into your lib folder. For me, that was

$ cp /var/lib/gems/1.9.1/gems/nanoc-3.7.5/lib/nanoc/helpers/blogging.rb lib/

It should be included like this in the lib/default.rb

include Nanoc3::Helpers::Blogging

Add the following attribute to the AtomFeedBuilder class

attr_accessor :format

If you don’t trust yourself to always remember the files you might want to  add this exception to the validate_feed_item function

if format.nil?
raise Nanoc::Errors::GenericTrivial.new('Cannot build Atom feed: no format(mp3,opus,blog) in params, item or site config')
end

After the # Add link comment is a good place to insert the  enclosure tag. File.size() will only work if the files are there and the exact same name. This code could probably be written a bit more safely, but I’m not a ruby developer and since I will have an mp3 file and and opus file in every post it’s not a problem this way.

# Add podcast enclosure
if format == 'mp3'
xml.link(href:"http://yourpodcast.com/mp3/" + a[:mp3],length:File.size("content/mp3/" + a[:mp3]), type:"audio/mpeg", rel:"enclosure")
elsif format == 'opus'
xml.link(href:"http://yourpodcast.com/opus/" + a[:opus],length:File.size("content/opus/" + a[:opus]), type:"audio/mpeg", rel:"enclosure")
end

To interpret the mp3 and opus attribute from earlier in the actual post, the last step is to add this line to the atom_feed function:

      builder.format            = params[:format]

You might need to install builder to let this run

$ sudo gem install builder

The only thing left to do is to edit the Rules file:

compile '/blogfeed' do
filter :erb
end
compile '/mp3feed' do
filter :erb
end
compile '/opusfeed' do
filter :erb
end
[...]
route '/blogfeed' do
'/blogfeed.xml'
end
route '/mp3feed' do
'/mp3feed.xml'
end
route '/opusfeed' do
'/opusfeed.xml'
end

 

You can find the blogging.rb and the Rules file on GitHub.

Flattr this!

Morsecode As A Service: node.js app with restify and Heroku

Posted on 22. Januar 2015 Comments

I wanted to play around with node.js and REST APIs. Heroku is widely used for deploying node.js app, last but not least because they give you one free instance to test your code and the possibility to use your own domain name (via CNAME).

Until now the code is rather trivial. This for example is the encode function (plaintext->morsecode). It uses the npm module morse to encode the given string in the request parameters and returns it with the plaintext in an array. The requests are handled by restify.
function encode(req, res, next) {
var answer = {}
answer.plaintext = req.params.string.toUpperString();
answer.morsecode = morse.encode(req.params.string);
res.send(answer);
next();
}

This function gets called later here.
server.get('/encode/:string', app.encode);

This starts the server on the port from the Heroku instance.
var port = process.env.PORT || 8080;
server.listen(port, function() {
console.log('%s listening at %s', server.name, server.url);
});

The decode function is equivalent, except that I test if there are only „.“, „-“ and white spaces in the request.

I wrote the documentation with the automatic page generator from GitHub Pages in the repository and so with the following code the user is redirected there when entering the root „/“.

 

function redirectToDocumentation(req,res,next) {
res.send(302, null, {
Location: API_DOKU
});
next();
}

It’s reachable under morsecode-api.de or morsecodeapi.herokuapp.com.

Flattr this!

Using a Perl script because refactoring of a project with Android tools didn’t work

Posted on 20. Dezember 2014 Comments

I wanted to rename my project but I guess since it has a lot of dependencies that caused an error somewhere and I got the error message:

Refactoring
A fatal error occurred while performing the refactoring.
An unexpected exception occurred while creating a change object. See the error log for more details.

So I just the normal refactoring feature of Eclipse which not surprisingly also caused an error. After editing the AndroidManifest package entry, the import of the resources in the sources files didn’t work. It still said

import com.example.oldpackage.R

Only a couple of resource files needed manual editing but the Java files were a problem. What did the trick for me was this one-liner

perl -pi -w -e 's/import com.example.oldpackage.R/import com.example.newpackage.R/g;' `grep -r -l "import com.example.oldpackage.R"`

I realise this is rather quick&dirty (do a backup 😉 ) but it did work for this project. A short explanation:

perl

  • -pi puts the code in a loop (like -n, but sed-style)
  • -w gives you warnings
  • -e is one line of programm and since -pi
  • /g global, for all lines in the file

grep

  • -r recursive
  • -l give out files that matches the following search string

Flattr this!

Automize selling at LaRedoute #2: Parse order files

Posted on 16. Dezember 2014 Comments

This blog post is part of the series Automize selling at LaRedoute.

  • Part 1: Get new orders
  • Part 2: Parse order files
  • Part 3: Upload response files
  • Part 4: update quantity and price feed

Update 2016: La Redoute is going to stop using CSV and moves everything to SOAP Webservices. Tutorials will follow


In part 1 the files containing orders are downloaded into a folder called OrdersFromLaRedoute. The next script is going to go through that folder, parse the file and insert it into a table.

These are the values the table must have because we’re just going to insert everything that’s in the CSV files.


$dbValues = ['MarketplaceID', 'OrderID', 'StorefrontOrderID', 'OrderDate', 'BuyerEmailAddress', 'BuyerName', 'BuyerPhoneNumber', 'OrderItemCode', 'ItemStatus', 'SKU', 'Title', 'Quantity', 'ItemPrice', 'ItemTax', 'ShippingCharge', 'ShippingTax','ItemFee', 'Currency', 'ShippingOption', 'PaymentInfo', 'ShippingAddressName', 'ShippingAddressFieldOne', 'ShippingAddressFieldTwo', 'ShippingAddressFieldThree', 'ShippingCity', 'ShippingStateOrRegion', 'ShippingPostalCode', 'ShippingCountryCode', 'ShippingPhoneNumber', 'BillingAddressName', 'BillingAddressFieldOne', 'BillingAddressFieldTwo', 'BillingAddressFieldThree', 'BillingCity', 'BillingStateOrRegion', 'BillingPostalCode', 'BillingCountryCode', 'BillingPhoneNumber'];

 

Therefore I created the table more or less like this:

CREATE TABLE `ORDERS-HISTORY` (
`MarketplaceID` int(11) DEFAULT NULL,
`OrderID` varchar(50) DEFAULT NULL,
`StorefrontOrderID` varchar(50) DEFAULT NULL,
`OrderDate` varchar(50) DEFAULT NULL,
`BuyerEmailAddress` varchar(50) DEFAULT NULL,
`BuyerName` varchar(50) DEFAULT NULL,
`BuyerPhoneNumber` varchar(50) DEFAULT NULL,
`OrderItemCode` varchar(50) NOT NULL DEFAULT '',
`ItemStatus` varchar(10) NOT NULL DEFAULT '',
`SKU` int(11) DEFAULT NULL,
`Title` varchar(50) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`ItemPrice` decimal(8,2) DEFAULT NULL,
`ItemTax` decimal(8,2) DEFAULT NULL,
`ShippingCharge` decimal(8,2) DEFAULT NULL,
`ShippingTax` decimal(8,2) DEFAULT NULL,
`ItemFee` decimal(8,2) DEFAULT NULL,
`Currency` varchar(3) DEFAULT NULL,
`ShippingOption` varchar(10) DEFAULT NULL,
`PaymentInfo` varchar(50) DEFAULT NULL,
`ShippingAddressName` varchar(70) DEFAULT NULL,
`ShippingAddressFieldOne` varchar(70) DEFAULT NULL,
`ShippingAddressFieldTwo` varchar(70) DEFAULT NULL,
`ShippingAddressFieldThree` varchar(70) DEFAULT NULL,
`ShippingCity` varchar(70) DEFAULT NULL,
`ShippingStateOrRegion` varchar(70) DEFAULT NULL,
`ShippingPostalCode` int(11) DEFAULT NULL,
`ShippingCountryCode` varchar(3) DEFAULT NULL,
`ShippingPhoneNumber` varchar(50) DEFAULT NULL,
`BillingAddressName` varchar(70) DEFAULT NULL,
`BillingAddressFieldOne` varchar(70) DEFAULT NULL,
`BillingAddressFieldTwo` varchar(70) DEFAULT NULL,
`BillingAddressFieldThree` varchar(70) DEFAULT NULL,
`BillingCity` varchar(70) DEFAULT NULL,
`BillingStateOrRegion` varchar(70) DEFAULT NULL,
`BillingPostalCode` int(11) DEFAULT NULL,
`BillingCountryCode` varchar(3) DEFAULT NULL,
`BillingPhoneNumber` varchar(50) DEFAULT NULL,
PRIMARY KEY (`OrderID`,`OrderItemCode`,`ItemStatus`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 

As you can see, the primary key consists of OrderID, OrderItemCode and ItemStatus. An OrderID is the unique identifier of one order, consisting of possibly many but at least one OrderItemCode. An OrderItemCode is representing an SKU + Quantity. For newly created orders, the ItemStatus will appear as „Created“. Once an item is accepted, LaRedoute will put a file into the ToSupplier folder with exactly the same OrderID, OrderItemCode but the ItemStatus „ToShip“. This will be important in Step 3.

Then, use a CSV library like league/csv and insert it with e.g. medoo.


$dirAsArray = scandir("OrdersFromLaRedoute");

foreach($dirAsArray as $file) {
// parse and INSERT
}

Flattr this!

Automize selling at LaRedoute #1: Get new orders

Posted on 16. Dezember 2014 Comments

This blog post is part of the series Automize selling at LaRedoute.


The french marketplace LaRedoute unfortunately doesn’t have a real API, but they do have ways to automize some processes. A lot of smaller marketplaces have this concept as well. You will get credentials for an SFTP server. On this server you will find the folders ToSupplier and FromSupplier, where the „supplier“ (aka you) can up- and download a range files documented by Merchantry in their blog. The processing of the uploaded files can take up to 6 hours, but is sometimes done in only a couple of minutes, so I’m going to assume the worst case of 6 hours in this post.

While programming a couple of scripts I found the following problems:

  • the server is incredibly slow sometimes (better at nights), so sometimes the connections just time out
  • sometimes the listing for the ToSupplier folder times out because there are too many files (according to support…huh?), so they have to be deleted regularly
  • not only the connection to LaRedoute but also the connection to my local MySQL server times out
  • I have to reserve a purchased item once I accepted it on LaRedoute immediately, because it could be sold elsewhere in the 6 hours LaRedoute might take to give me the shipping address

New orders can be found in the ToSupplier folder in tab seperated CSV files (but .txt ending) with the format OrdersYYYY-MM-DD-hh-mm-ss.txt.

Since PHP is the companies main language I will show a couple of scripts which automize downloading and processing those files. The code is of course simplified for better understanding. We’re using SFTP instead of FTP and I found using the phpseclib to be the most usable library.

I will propose the use of 2 Tables in the MySQL database: TEMP-FILENAMES and FILENAMES-HISTORY. Both have a the unique column filename. FILENAMES-HISTORY will contain the name of every file ever processed by the following script, TEMP-FILENAMES is a helper table that will be truncated after every run.

First, we need to establish a connection


$sftp = new Net_SFTP(SFTP_LAREDOUTE_HOST);
if (!$sftp-&gt;login(SFTP_LAREDOUTE_USER, SFTP_LAREDOUTE_PASS)) {
exit('Login Failed');
}

Then we change directory. This is a command that usually involves listing the directory changed into, but since this is not a graphical client, the real timeout might come on line below. $nlist will just be null if the listing fails, and I will assume it didn’t work if it takes more than 30 seconds.

$sftp->chdir('/ToSupplier');

$beforetime = time();
$nlist = $sftp->nlist();
$aftertime = time();
if(($aftertime-$beforetime) > 30 ) {
exit('Timeout while Listing directory');
}

The next piece of code is only executed if the listing worked. Every filename that includes the word „Order“ is now inserted into the temporary table:

foreach($nlist as $filename) {
if (strpos($filename, 'Order') !== false) {
$qry = "INSERT INTO `TEMP-FILENAMES`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry,MYSQLCONNECTION) or print mysql_error();
}
}

You can look at the difference between the filenames in your HISTORY table and the possibly new ones in the temporary table.

$tmpCmpFilenames = array();
$qry = "SELECT `filename` FROM `TEMP-FILENAMES` WHERE `filename` NOT IN (SELECT `filename` FROM `FILENAMES-HISTORY`)";
$select = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();
while ($row = mysql_fetch_assoc($select)) {
$tmpCmpFilenames[] = $row['filename-id'];
}

Now we have all the new files in the array $tmpCmpFilenames. The correct way would be make sure the downloaded files are correct with hashes. Instead we decided to misuse the filesize, since it’s a good indicator something didn’t work properly;) The files not downloaded correctly are deleted from the array. They will appear next time the script is run.

foreach($tmpCmpFilenames as $filename) {
$remotefilesize = $sftp->size($filename);
$sftp->get($filename, 'OrdersFromLaRedoute/' . $filename);
$localfilesize = filesize('OrdersFromLaRedoute/' . $filename);
if ($remotefilesize != $localfilesize) {
unset($tmpCmpFilenames[$filename]);
}
}

We can now insert the filenames into the HISTORY table.

foreach($tmpCmpFilenames as $filename) {
$qry = "INSERT INTO `FILENAMES-HISTORY`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();
}

Last but not least, the temporary table needs to be truncated for the next run.

$truncate=mysql_query("TRUNCATE TABLE `TEMP-FILENAMES`",MYSQLCONNECTION) or print mysql_error();

The next step is described in part 2 of this series.

Flattr this!