Entries filed under English

Automize selling at LaRedoute #1: Get new orders

Posted on 16. Dezember 2014 Comments

This blog post is part of the series Automize selling at LaRedoute.

The french marketplace LaRedoute unfortunately doesn’t have a real API, but they do have ways to automize some processes. A lot of smaller marketplaces have this concept as well. You will get credentials for an SFTP server. On this server you will find the folders ToSupplier and FromSupplier, where the „supplier“ (aka you) can up- and download a range files documented by Merchantry in their blog. The processing of the uploaded files can take up to 6 hours, but is sometimes done in only a couple of minutes, so I’m going to assume the worst case of 6 hours in this post.

While programming a couple of scripts I found the following problems:

  • the server is incredibly slow sometimes (better at nights), so sometimes the connections just time out
  • sometimes the listing for the ToSupplier folder times out because there are too many files (according to support…huh?), so they have to be deleted regularly
  • not only the connection to LaRedoute but also the connection to my local MySQL server times out
  • I have to reserve a purchased item once I accepted it on LaRedoute immediately, because it could be sold elsewhere in the 6 hours LaRedoute might take to give me the shipping address

New orders can be found in the ToSupplier folder in tab seperated CSV files (but .txt ending) with the format OrdersYYYY-MM-DD-hh-mm-ss.txt.

Since PHP is the companies main language I will show a couple of scripts which automize downloading and processing those files. The code is of course simplified for better understanding. We’re using SFTP instead of FTP and I found using the phpseclib to be the most usable library.

I will propose the use of 2 Tables in the MySQL database: TEMP-FILENAMES and FILENAMES-HISTORY. Both have a the unique column filename. FILENAMES-HISTORY will contain the name of every file ever processed by the following script, TEMP-FILENAMES is a helper table that will be truncated after every run.

First, we need to establish a connection

exit('Login Failed');

Then we change directory. This is a command that usually involves listing the directory changed into, but since this is not a graphical client, the real timeout might come on line below. $nlist will just be null if the listing fails, and I will assume it didn’t work if it takes more than 30 seconds.


$beforetime = time();
$nlist = $sftp->nlist();
$aftertime = time();
if(($aftertime-$beforetime) > 30 ) {
exit('Timeout while Listing directory');

The next piece of code is only executed if the listing worked. Every filename that includes the word „Order“ is now inserted into the temporary table:

foreach($nlist as $filename) {
if (strpos($filename, 'Order') !== false) {
$qry = "INSERT INTO `TEMP-FILENAMES`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry,MYSQLCONNECTION) or print mysql_error();

You can look at the difference between the filenames in your HISTORY table and the possibly new ones in the temporary table.

$tmpCmpFilenames = array();
$qry = "SELECT `filename` FROM `TEMP-FILENAMES` WHERE `filename` NOT IN (SELECT `filename` FROM `FILENAMES-HISTORY`)";
$select = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();
while ($row = mysql_fetch_assoc($select)) {
$tmpCmpFilenames[] = $row['filename-id'];

Now we have all the new files in the array $tmpCmpFilenames. The correct way would be make sure the downloaded files are correct with hashes. Instead we decided to misuse the filesize, since it’s a good indicator something didn’t work properly;) The files not downloaded correctly are deleted from the array. They will appear next time the script is run.

foreach($tmpCmpFilenames as $filename) {
$remotefilesize = $sftp->size($filename);
$sftp->get($filename, 'OrdersFromLaRedoute/' . $filename);
$localfilesize = filesize('OrdersFromLaRedoute/' . $filename);
if ($remotefilesize != $localfilesize) {

We can now insert the filenames into the HISTORY table.

foreach($tmpCmpFilenames as $filename) {
$qry = "INSERT INTO `FILENAMES-HISTORY`(`filename`) VALUES ('". $filename . "')";
$insert = mysql_query($qry, MYSQLCONNECTION) or print mysql_error();

Last but not least, the temporary table needs to be truncated for the next run.

$truncate=mysql_query("TRUNCATE TABLE `TEMP-FILENAMES`",MYSQLCONNECTION) or print mysql_error();

The next step is described in part 2 of this series.

Flattr this!

Minimal Mensa Plan

Posted on 9. Dezember 2014 Comments

We’ve been doing a lot of website scraping for a university project latel so I decided a little app to scrape the universitys‘ cantine website for the lunch menu.


A network connection in the main Activity is not allowed so I’m using a private class for that. Once the doInBackground method is over onPostExecute is automatically called.

Luckily the jSoup library which is used for parsing also brings a way download a website for parsing.

Document doc = Jsoup.connect("http://speiseplan.studierendenwerk-hamburg.de/de/520/2014/0/").get();

There is only one category class and it contains the date. The dish-description class contains what you will later see in the app. The size is needed for a for-loop later on.

date = doc.select(".category").text();
int maxDishes = doc.getElementsByClass("dish-description").size();

With the next piece of code all the dish descriptions are extracted. The original website contains details about the food (made with alcohol, pork or if it’s vegeterian etc). For blending this out I use a regular expression which filters out:

1 non-word character(\W = an opening bracket), then possibly multiple digits (\d) and non-word characters (\W = commas) and then another non-word character (\W = a closing bracket)

String dish = doc.getElementsByClass("dish-description").get(i).text().replaceAll("(\\W[\\d\\W]*\\W)", " ");

These dishes are then saved together with every second price (the one for students, the other one’s for employees) in a HashMap, which is then added to a list. For recognizing later, the dishes get the key „dish“ and the prices the key „price“.

Once the data extraction is done, onPostExecute is automatically called. The date is set to a TextView above the ListView and a SimpleAdapter is populating the list of HashMaps into the layout

simpleAdapter = new SimpleAdapter(MainActivity.this,
dishList, R.layout.list, new String[] { "dish",
"price" }, new int[] { R.id.text1, R.id.text2 });

Since i’s called Minimal Mensa Plan, no other features (such as caching or selecting a cantine) are available. The app in the Play Store is used for scraping the cantine at the campus Berliner Tor but it might as well be used for others, just by changing the URL. It’s released under the MIT License and available at GitHub.

Flattr this!

How to disable ACK with xbee-arduino API

Posted on 23. April 2014 Comments

It might be useful to disable all the ACK packages(0x8B) for testing purposes. That’s possible with all packages that inherit from XBeeRequest, my example shows a ZBTxRequest.


Flattr this!

Markdown Templates for README files on GitHub

Posted on 7. März 2014 Comments

Every time I upload a new project on GitHub I find myself writing almost an identicially structured README.md file in Markdown. So I thought I could make some templates for the next times. Right now I only have one for mobile apps and websites with GitHub Pages, but there’s more to come.

Also I found that quite a lot of projects have very unstructured and ugly READMEs. It’s always nice to get a quick overview over a project, especially when they don’t have a website. So maybe someone can use this.

Flattr this!

10 best Android apps for the paranoid

Posted on 15. Dezember 2013 Comments

So I compiled a few apps that even the most paranoid people would feel safe to use. Privacy is mostly enforced through encryption but beware: if you don’t use encryption correctly by e.g. chosing a weak password, it’s totally useless. You can download from Google Play (GP) or F-Droid (FD)


TextSecure (GP)

TextSecure encrypts your text messages(SMS) locally. Also, if your partner also has TextSecure, you can encrypt the messages, so not even the carrier could intercept them.

update: No SMS encryption functionality anymore, but this app is now called Signal and can still encrypt the SMS and message database on the phone. For SMS encryption use Silence (GP).


Cost: free

RedPhone (GP)

RedPhone is basically an open source Skype replacement with even better encryption. It’s dead simple to use. If a contact in your phonebook also uses RedPhone, you will be asked if you want to upgrade to a secure call.

update: Now merged with TextSecure in the Signal Messenger App.


Cost: free


Google Authenticator (GP)

Whether it’s your bank, your WordPress blog, github, Dropbox or facebook. Two factor authentication with One Time Passwords(OTPs) is getting popular! With this app, you will not only need your password to login, but a code from your smartphone that’s generated and valid for 30 seconds. Just in case your phone gets stolen or lost, make sure to print out extra backup code. Don’t want them laying around your house? Use the next app!


Cost: free


EDSLite (GP)

EDSLite works much like Truecrypt/Veracrypt for your computer. You can create containers, where you can store your Pins, Tans, Backup OTP Backup Codes and more. Because TrueCrypt can read those containers you can just save them to your Dropbox and keep them in sync with your computer. If you don’t trust Dropbox, maybe the next app is for you.

update: Use Veracrypt instead of Truecrypt, since Truecrypt is not actively developed anymore. EDS can use both.


Cost: free


ownCloud (GP/FD)

ownCloud is an app that works with your owncloud installation. You can download your private cloud and host it from home on a Rapberry Pi or chose a hosted solution in a data center that you trust.


Cost: 0.79€



So, from where should you download all these apps? Trust Google or Amazon that the binaries you’re getting are what the developers uploaded? f-droid.org is another alternative app store that you might consider to trust. The apps are all open source and you can probably even get a couple of apps for free that would cost a few cents in the Google Play Store.

Cost: free


Threema (GP/Website)

Threema is probably the best app available for secure encrypted messaging; it works very much like Whatsapp. The servers are in Switzerland. The only downturn is: it’s not open source. But: you can verify that the encryption works correctly. If you just add a contact with his/her Threema ID, the contact appears read. If you sync your phonebook with Threema(your phone number is only transmitted as a hash) and Threema recognizes a contact, it’s getting orange. And if you meet your contact in person and scan his QR-code, the contact finally turns green. This is the most paranoid messaging app I’ve seen.


Cost: 1.60€


k-9 Mail (GP/FD)

If you want to send only encrypted emails, this is for you. Together with the APG app, it’s super easy to send encrypted and receive/decrypt emails.


Cost: free


Built-in Android encryption

With Android 4.x on you can encrypt your whole device. Make sure you chose and remember a good password or PIN.

Cost: free


NoteCipher (GP/FD)

Evernote is great for keeping your notes in sync. But if you want to keep them secret, you might want to consider using this app. Every entry is encrypted via sqlcipher with AES 256.


Cost: free


More stuff

  • Heml.is looks promising but has been under development for quite some time now. Hope to hear more soon.
    • Heml.is is dead.
  • Telegram is another open source messaging app. The encryption only works, when both participants are online, so be careful.
  • Droid-Break has a lot more high quality open source apps


GP: Google Play, FD: F-Droid, Pictures from play.google.com

Flattr this!

Show Pictures in a Form on a Table with Report in APEX

Posted on 10. Dezember 2013 Comments

The table is called WPDBD_TEIL. The primary key is TID. In WPDBD_TEIL there is a column called BILD(german: picture/image) from type BLOB.

First click on your form and keep the name of the picture in mind. In this case, it’s P9_BILD.

Then go to your report, click on edit for the whole report and use the following code


select "TID", 
decode(nvl(dbms_lob.getlength("BILD"),0),0,null, '<img height="100px" width="100px" src="'||apex_util.get_blob_file_src('P9_BILD',TID)||'" />') "BILD"

If you’d open the report now, you only get the HTML Code. So you have to change the way APEX displays the entry. So go to the report, click on BILD and change Display as Text to Report Standard Column




The form I used was the following.


Also, in the last page of creating the form, you have to chose your own primary key, not the ROWID.




Flattr this!

How to compile async for scala with sbt

Posted on 6. Dezember 2013 Comments

SBT is the build tool for Scala, much like Maven or Ant is for Java. Scalas async library uses it for compiling. You could just it from the source or compile it into a handy .jar file. That’s where SBT comes in. The Scala version I used it 2.10.3, SBT in version 0.13.0 and async in 0.9.0. The build.sh in the repository didn’t work for me.

Clone the lastest version of the GitHub repository: https://github.com/scala/async

git clone https://github.com/scala/async.git

Then download and install sbt from scala-sbt.org. For Ubuntu 12.04 I just used the .deb package.

wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.0/sbt.deb

And install it.

sudo dpkg -i sbt.deb

Then open it in the same folder you cloned the git repository:

repat@laptop:~/asynctest/async$ sbt
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
[info] Loading project definition from /home/repat/asynctest/async/project
[info] Updating {file:/home/repat/asynctest/async/project/}async-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to scala-async (in build file:/home/repat/asynctest/async/)

Maybe a couple of updates will appear here and depending on your internet connection it might actually take a while.
Once you get the > sign type compile and this will happen:

[info] Updating {file:/home/repat/asynctest/async/}async...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 15 Scala sources to /home/repat/asynctest/async/target/scala-2.10/classes...
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] one warning found
[success] Total time: 28 s, completed 06.12.2013 19:22:09

You will now have the compiled .class files in …/async/target/scala-2.10. The get the .jar do the following.

> publish
[info] Packaging /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT-sources.jar ...
[info] Done packaging.
[info] Wrote /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT.pom
[info] :: delivering :: org.scala-lang.modules#scala-async_2.10;0.9.0-SNAPSHOT :: 0.9.0-SNAPSHOT :: integration :: Fri Dec 06 19:25:16 CET 2013
[info] 	delivering ivy file to /home/repat/asynctest/async/target/scala-2.10/ivy-0.9.0-SNAPSHOT.xml
[info] Main Scala API documentation to /home/repat/asynctest/async/target/scala-2.10/api...
[warn] there were 2 feature warning(s); re-run with -feature for details
model contains 29 documentable templates
[warn] /home/repat/asynctest/async/src/main/scala/scala/async/Async.scala:11: Could not find any member to link for "scala.concurrent.Future".
[warn] /**
[warn] ^
[warn] /home/repat/asynctest/async/src/main/scala/scala/async/internal/FutureSystem.scala:9: Could not find any member to link for "scala.async.AsyncBase".
[warn] /**
[warn] ^
[warn] three warnings found
[info] Main Scala API documentation successful.
[info] Packaging /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[trace] Stack trace suppressed: run last *:publish for the full output.
[error] (*:publish) java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized
[error] Total time: 12 s, completed 06.12.2013 19:25:27

I’m not sure why it stops with an error here or rather, why access is denied to that repository. Anyway, the scala-async_2.10-0.9.0-SNAPSHOT.jar file is now in …/async/target/scala-2.10.

As mentioned in the error message, you get the stacktrace with last *:publish. I leave this here so people can find it via google.

last *:publish 
java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized
	at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:79)
	at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:231)
	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)
	at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:84)
	at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
	at org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:234)
	at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:216)
	at sbt.IvyActions$$anonfun$publish$3.apply(IvyActions.scala:258)
	at sbt.IvyActions$$anonfun$publish$3.apply(IvyActions.scala:257)
	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
	at sbt.IvyActions$.publish(IvyActions.scala:257)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply$mcV$sp(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$.withChecksums(IvyActions.scala:102)
	at sbt.IvyActions$.sbt$IvyActions$$withChecksums(IvyActions.scala:97)
	at sbt.IvyActions$$anonfun$publish$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1.apply(IvyActions.scala:87)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
	at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
	at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
	at xsbt.boot.Using$.withResource(Using.scala:11)
	at xsbt.boot.Using$.apply(Using.scala:10)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
	at sbt.IvySbt.withIvy(Ivy.scala:101)
	at sbt.IvySbt.withIvy(Ivy.scala:97)
	at sbt.IvySbt$Module.withModule(Ivy.scala:116)
	at sbt.IvyActions$.publish(IvyActions.scala:87)
	at sbt.Classpaths$$anonfun$publishTask$1.apply(Defaults.scala:1094)
	at sbt.Classpaths$$anonfun$publishTask$1.apply(Defaults.scala:1093)
	at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:35)
	at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:34)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
	at sbt.std.Transform$$anon$4.work(System.scala:64)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
	at sbt.Execute.work(Execute.scala:244)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
[error] (*:publish) java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized

Flattr this!

Updating Owncloud News App with cron and Uberspace

Posted on 27. November 2013 Comments

Since it’s not possible to create a cronjob with the user the webserver is running under(as advised), I just created a cronjob with my user(crontab -e) and it works just fine.

[repat@uberspace]$ crontab -l
*/15 * * * * /package/host/localhost/php/bin/php -f /home/repat/html/owncloud/cron.php

The problem was the following: owncloud wouldn’t update my news and I didn’t think it was a cron problem, since I couldn’t update it manually(AJAX). Adding new feeds however did work. I still don’t get why manual updating doesn’t work , even with the Android app. Opening cron.php in the webbrowser didn’t work, although it did display {„status“:“success“}..turns out: it has to be called via CLI.

Unfortunately, I get an email every 15 minutes now… To disable that I added this above the cron entry:


Thanks to Raydiation and j-ed for helping me.

Flattr this!

„Incomplete statements not supported yet, sorry, you’ll have to retype“ Error with Scala and Eclipse

Posted on 15. November 2013 Comments

I got this error defining the function

object test {
  val f:AnyVal => String = {
      case 10.0 => "10.0"
      case _: Double => "Double"
      case _: Int => "Int"
      case _: Byte => "Byte"
      case _ => "anything else"

 -> Incomplete statements not supported yet, sorry, you’ll have to retype.

However, in REPL it worked just fine.

repat@laptop:~$ scala
Welcome to Scala version 2.10.3 (OpenJDK 64-Bit Server VM, Java 1.6.0_27).
Type in expressions to have them evaluated.
Type :help for more information.

scala>   val f:AnyVal => String = {
     |       case 10.0 => "10.0"
     |       case _: Double => "Double"
     |       case _: Int => "Int"
     |       case _: Byte => "Byte"
     |       case _ => "anything else"
     |   }
f: AnyVal => String = 


So aparrently the error is in Eclipse. I use the version 3.7.2. Have you tried turning it off and on again? 😉

Flattr this!