Entries filed under Misc Scripting

Delete empty rows in CSV file with sed

Posted on 23. August 2017 Comments

This is for when you have empty lines (so lot’s of ,,,,,,) in your file.


sed -i '' -e '/^,*$/d' filename.csv
find ./ -type f -exec sed -i '' -e '/^,*$/d' {} \;


sed -i '/^,*$/d' filename.csv
find ./ -type f -exec sed -i '/^,*$/d' {} \;


^ marks the beginning of a line
,* marks a possible infinite amount of commas
$ marks the end of a line

Flattr this!

Simple wget crawler for list of files

Posted on 6. August 2015 Comments

This script could be helpful to download a set of files from a webserver, that you don’t have (S)FTP access to. The input file consists of a list of filenames, one name each line.

while IFS= read -r line; do
wget -nc -R --spider $FULLURL
done < "$file"

At first, the first command line argument is saved into the variable file. Then the Webserver address is saved to the WEBSERVER variable. IFS stands for Internal Field Separator. It’s used to read line by line through the file in the while loop that ends in the last line. Inside of the loop, the read line is concatenated with the webserver address into FULLURL. Then, wget is used with the parameters -nc for checking if the file is not already present in the current folder, -R for downloading and –spider for checking the existence on the webserver.

You can find the script on GitHub.

Flattr this!

Using nanoc for podcast feeds

Posted on 23. Juli 2015 Comments

Nanoc is a static site generator much like jekyll or octopress, but with a more minimalistic approach. These generators are not necessarily the most suitable choice for a podcast website, but it’s possible and you might save up on webspace and traffic when you use GitHub or Neocities.

Creating the podcast feed is basically like writing a normal Atom feed for the blog, since podcast feeds ARE indeed feeds with an enclosure tag in which the URL to the audio or video file is placed. This guide does not include the itunes tags. I might add it one day. Follow the instructions and use the documentation for the Helper Blogging. Tag your podcast episodes as kind:article.

The media files are placed in content/mp3 and content/opus, which is where the links in the feeds will point to later.

I invented the fields mp3 and opus, since these are the file formats I want to use. The values are the filenames. The header of a new episode/post would look like

title: 001 - Podcast Episode Title
created_at: 2015-03-14 09:00:00 +0000
kind: article
tags: [podcast,topic]
mp3: 001-podcast-episode-title.mp3
opus: 001-podcast-episode-title.opus

This has to be filled manually everytime, so make sure you have the exact filename, as some podcast clients won’t allow correction of the URL.

The next step is to write different feeds for the formats. For that, I’m using the new field format, which will be interpreted by the Helper class later on. For example, I called my normal feed blogfeed and the podcast feeds mp3feed and opusfeed. Create the file blogfeed.erb in the content folder and fill it with the following:

<%= atom_feed :title => 'repats podcast blog', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'blog' %>

The mp3feed.erb and opusfeed.erb are filled accordingly:

<%= atom_feed :title => 'repats podcast mp3', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'mp3' %>

<%= atom_feed :title => 'repats podcast opus', :author_name => 'repat',
:author_uri => 'http://repat.de', :limit => 10, :format => 'opus' %>

The next step is to use the Blogging locally in your nanoc installation. To do that you need to copy it from the gems folder into your lib folder. For me, that was

$ cp /var/lib/gems/1.9.1/gems/nanoc-3.7.5/lib/nanoc/helpers/blogging.rb lib/

It should be included like this in the lib/default.rb

include Nanoc3::Helpers::Blogging

Add the following attribute to the AtomFeedBuilder class

attr_accessor :format

If you don’t trust yourself to always remember the files you might want to  add this exception to the validate_feed_item function

if format.nil?
raise Nanoc::Errors::GenericTrivial.new('Cannot build Atom feed: no format(mp3,opus,blog) in params, item or site config')

After the # Add link comment is a good place to insert the  enclosure tag. File.size() will only work if the files are there and the exact same name. This code could probably be written a bit more safely, but I’m not a ruby developer and since I will have an mp3 file and and opus file in every post it’s not a problem this way.

# Add podcast enclosure
if format == 'mp3'
xml.link(href:"http://yourpodcast.com/mp3/" + a[:mp3],length:File.size("content/mp3/" + a[:mp3]), type:"audio/mpeg", rel:"enclosure")
elsif format == 'opus'
xml.link(href:"http://yourpodcast.com/opus/" + a[:opus],length:File.size("content/opus/" + a[:opus]), type:"audio/mpeg", rel:"enclosure")

To interpret the mp3 and opus attribute from earlier in the actual post, the last step is to add this line to the atom_feed function:

      builder.format            = params[:format]

You might need to install builder to let this run

$ sudo gem install builder

The only thing left to do is to edit the Rules file:

compile '/blogfeed' do
filter :erb
compile '/mp3feed' do
filter :erb
compile '/opusfeed' do
filter :erb
route '/blogfeed' do
route '/mp3feed' do
route '/opusfeed' do


You can find the blogging.rb and the Rules file on GitHub.

Flattr this!

Morsecode As A Service: node.js app with restify and Heroku

Posted on 22. Januar 2015 Comments

I wanted to play around with node.js and REST APIs. Heroku is widely used for deploying node.js app, last but not least because they give you one free instance to test your code and the possibility to use your own domain name (via CNAME).

Until now the code is rather trivial. This for example is the encode function (plaintext->morsecode). It uses the npm module morse to encode the given string in the request parameters and returns it with the plaintext in an array. The requests are handled by restify.
function encode(req, res, next) {
var answer = {}
answer.plaintext = req.params.string.toUpperString();
answer.morsecode = morse.encode(req.params.string);

This function gets called later here.
server.get('/encode/:string', app.encode);

This starts the server on the port from the Heroku instance.
var port = process.env.PORT || 8080;
server.listen(port, function() {
console.log('%s listening at %s', server.name, server.url);

The decode function is equivalent, except that I test if there are only „.“, „-“ and white spaces in the request.

I wrote the documentation with the automatic page generator from GitHub Pages in the repository and so with the following code the user is redirected there when entering the root „/“.


function redirectToDocumentation(req,res,next) {
res.send(302, null, {
Location: API_DOKU

It’s reachable under morsecode-api.de or morsecodeapi.herokuapp.com.

Flattr this!

How to change nano syntax highlighting for arbitrary filename extensions

Posted on 17. Dezember 2014 Comments

The text editor nano manages syntax highlighting through .nanorc files. Say, we have files with the ending .phpx, but would like PHP syntax highlighting.

$ sudo nano /usr/share/nano/php.nanorc

Change this line

syntax "php" "\.php[2345s~]?$"

to this regex:

syntax "php" "\.php[x2345s~]?$".

Alternativly, e.g. if you don’t have sudo rights for /usr/share/nano/php.nanorc, you can do the following

$cp /usr/share/nano/php.nanorc ~/.phpx.nanorc

Then enter the following line in ~/.nanorc:

include ~/.phpx.nanorc

Then, change the line as mentioned above and syntax „php“ into syntax „phpx“

Flattr this!

Alle norwegischen Sonderzeichen in Dateinamen ersetzen

Posted on 15. Juli 2014 Comments

Wenn Dateien von einem norwegischen PC stammen, könnte es sein, dass die Dateinamen nicht mit UTF-8 Zeichen geschrieben sind. Man sieht dann ein Fragezeichen Symbol bei der ls-Ausgabe. Dieser Weg ist alles andere als elegant, aber funktioniert, wenn es einfach nur schnell und vielleicht nur einmal überhaupt erledigt werden soll. Wenn das öfter vorkommt, wäre ein Skript angebrachter. Vom User Gilles von Stack Exchange habe ich dieses Skript kopiert:


grep-invalid-utf8 (){
  perl -l -ne '/^([\000-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277]{2}|[\360-\367][\200-\277]{3}|[\370-\373][\200-\277]{4}|[\374-\375][\200-\277]{5})*$/ or print'}
find | grep-invalid-utf8

Das kann man so einfach in der Konsole absetzen. Danach habe ich wie vorgeschlagen mit diesem Skript, find und rename alle Dateien UTF-8 Dateinamen gegeben.

find | grep-invalid-utf8 |
rename 'BEGIN {binmode STDIN, ":encoding(latin1)"; use Encode;}
        $_=encode("utf8", $_)' 

Jetzt habe ich mir mit ls > ls.txt alle Dateinamen in eine Textdatei geschrieben und diese in Libre Office Calc kopiert. In die A-Spalte und die B-Spalte kommen die Dateinamen. Dann wird auf die B-Spalte ein Makro angewandt, dass alle Sonderzeichen entfernt. Anschließend kommt in die C Spalte der folgende Befehl:

="mv "&A1&" "&B1

und in die D Spalte:


Jetzt kann man die D-Spalte kopieren und in ein Shellskript einfügen und im Ordner ausführen. Nicht vergessen das Skript mit chmod +x ausführbar zu machen.


Flattr this!

Libre Office Calc Makro zum Ersetzen von Sonderzeichen in BASIC

Posted on 22. Mai 2014 Comments

Dieses Makro ersetzt in einem LibreOffice Calc Sheet alle norwegischen Sonderzeichen (å, ø, æ) durch ASCII kompatible Zeichen.


    dim oSheet as object, mySuche as Object
    'erste Tabelle
    oSheet = thisComponent.Sheets.getByIndex(0)
    With mySuche
    .SearchRegularExpression = False
    end With
    With mySuche
    .SearchRegularExpression = False
    end With

    With mySuche
    .SearchRegularExpression = False
    end With
End Sub

Eine Anleitung, wie man das Makro in die Symbolleiste bekommt, findet sich im OpenOffice Wiki.

Flattr this!

Nullstellenberechnung in Lua

Posted on 16. Dezember 2013 Comments

Nachdem ich 2011 schonmal ein paar Methoden zur numerischen Nullstellenberechnung in MATLAB online gestellt hatte, hier nur dieselben Beispiele in Lua. Ich habe mit Lua gerade erst angefangen und eigentlich erstmal  zur Übung nur den Code übersetzt. Der Code ist natürlich auf GitHub. Eine gute Lua Referenz ist Learn Lua in 15 Minutes von @tylerneylon.

Wie im letzten Beispiel habe ich diese Funktion benutzt:

Außer bei Newton(da braucht man noch die Ableitung) sieht das dann in Lua so aus.

function f(x)
  return x^6-x-1

Zusätzlich habe ich noch die Genauigkeit Eta global festgesetzt:

eta = 0.00001


function bisection(a,b)
  local i = 0

  if (f(b)*f(a)<0) then
    while (math.abs(b-a)>eta) do
      if (f((a+b)/2)*f(a)>0) then
        a = ((a+b)/2)
        b = ((a+b)/2)
      i = i + 1
    return ((a+b)/2)


function newton(x)
  local i = 0
  local diff = 1
  while (diff>eta) do
    nullstelle = x
    t1, t2 = f(x)
    x = x - (t1/t2)
    i = i+1
    diff = math.abs(x-nullstelle)
  return x

function f(x)
  return x^6-x-1, 6*x^5-1

Regula Falsi

function regula_falsi(a,b)
  local i = 0
  local diff = 1
  if (f(b)*f(a)<0) then
    nullstelle = a
     while (diff > eta) do
      x = (a*f(b)-b*f(a))/(f(b)-f(a));
      if (f(x)<0) then
        a = x
        b = x
      diff = math.abs(x-nullstelle)
      nullstelle = x
      i = i+1
  return nullstelle


function secant(a,b)
  local i = 0
  while(math.abs(b-a)>eta) do
    nullstelle = b
    b = b-f(b)*(b-a)/(f(b)-f(a))
    a = nullstelle
    i = i+1
  return ((a+b)/2)

Flattr this!

How to compile async for scala with sbt

Posted on 6. Dezember 2013 Comments

SBT is the build tool for Scala, much like Maven or Ant is for Java. Scalas async library uses it for compiling. You could just it from the source or compile it into a handy .jar file. That’s where SBT comes in. The Scala version I used it 2.10.3, SBT in version 0.13.0 and async in 0.9.0. The build.sh in the repository didn’t work for me.

Clone the lastest version of the GitHub repository: https://github.com/scala/async

git clone https://github.com/scala/async.git

Then download and install sbt from scala-sbt.org. For Ubuntu 12.04 I just used the .deb package.

wget http://repo.scala-sbt.org/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.13.0/sbt.deb

And install it.

sudo dpkg -i sbt.deb

Then open it in the same folder you cloned the git repository:

repat@laptop:~/asynctest/async$ sbt
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
[info] Loading project definition from /home/repat/asynctest/async/project
[info] Updating {file:/home/repat/asynctest/async/project/}async-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to scala-async (in build file:/home/repat/asynctest/async/)

Maybe a couple of updates will appear here and depending on your internet connection it might actually take a while.
Once you get the > sign type compile and this will happen:

[info] Updating {file:/home/repat/asynctest/async/}async...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 15 Scala sources to /home/repat/asynctest/async/target/scala-2.10/classes...
[warn] there were 2 feature warning(s); re-run with -feature for details
[warn] one warning found
[success] Total time: 28 s, completed 06.12.2013 19:22:09

You will now have the compiled .class files in …/async/target/scala-2.10. The get the .jar do the following.

> publish
[info] Packaging /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT-sources.jar ...
[info] Done packaging.
[info] Wrote /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT.pom
[info] :: delivering :: org.scala-lang.modules#scala-async_2.10;0.9.0-SNAPSHOT :: 0.9.0-SNAPSHOT :: integration :: Fri Dec 06 19:25:16 CET 2013
[info] 	delivering ivy file to /home/repat/asynctest/async/target/scala-2.10/ivy-0.9.0-SNAPSHOT.xml
[info] Main Scala API documentation to /home/repat/asynctest/async/target/scala-2.10/api...
[warn] there were 2 feature warning(s); re-run with -feature for details
model contains 29 documentable templates
[warn] /home/repat/asynctest/async/src/main/scala/scala/async/Async.scala:11: Could not find any member to link for "scala.concurrent.Future".
[warn] /**
[warn] ^
[warn] /home/repat/asynctest/async/src/main/scala/scala/async/internal/FutureSystem.scala:9: Could not find any member to link for "scala.async.AsyncBase".
[warn] /**
[warn] ^
[warn] three warnings found
[info] Main Scala API documentation successful.
[info] Packaging /home/repat/asynctest/async/target/scala-2.10/scala-async_2.10-0.9.0-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[trace] Stack trace suppressed: run last *:publish for the full output.
[error] (*:publish) java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized
[error] Total time: 12 s, completed 06.12.2013 19:25:27

I’m not sure why it stops with an error here or rather, why access is denied to that repository. Anyway, the scala-async_2.10-0.9.0-SNAPSHOT.jar file is now in …/async/target/scala-2.10.

As mentioned in the error message, you get the stacktrace with last *:publish. I leave this here so people can find it via google.

last *:publish 
java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized
	at org.apache.ivy.util.url.AbstractURLHandler.validatePutStatusCode(AbstractURLHandler.java:79)
	at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:231)
	at org.apache.ivy.util.FileUtil.copy(FileUtil.java:150)
	at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:84)
	at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
	at org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:234)
	at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:216)
	at sbt.IvyActions$$anonfun$publish$3.apply(IvyActions.scala:258)
	at sbt.IvyActions$$anonfun$publish$3.apply(IvyActions.scala:257)
	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
	at sbt.IvyActions$.publish(IvyActions.scala:257)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply$mcV$sp(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1$$anonfun$apply$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$.withChecksums(IvyActions.scala:102)
	at sbt.IvyActions$.sbt$IvyActions$$withChecksums(IvyActions.scala:97)
	at sbt.IvyActions$$anonfun$publish$1.apply(IvyActions.scala:93)
	at sbt.IvyActions$$anonfun$publish$1.apply(IvyActions.scala:87)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
	at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
	at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:81)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
	at xsbt.boot.Using$.withResource(Using.scala:11)
	at xsbt.boot.Using$.apply(Using.scala:10)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
	at sbt.IvySbt.withIvy(Ivy.scala:101)
	at sbt.IvySbt.withIvy(Ivy.scala:97)
	at sbt.IvySbt$Module.withModule(Ivy.scala:116)
	at sbt.IvyActions$.publish(IvyActions.scala:87)
	at sbt.Classpaths$$anonfun$publishTask$1.apply(Defaults.scala:1094)
	at sbt.Classpaths$$anonfun$publishTask$1.apply(Defaults.scala:1093)
	at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:35)
	at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:34)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
	at sbt.std.Transform$$anon$4.work(System.scala:64)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
	at sbt.Execute.work(Execute.scala:244)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
[error] (*:publish) java.io.IOException: Access to URL https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/modules/scala-async_2.10/0.9.0-SNAPSHOT/scala-async_2.10-0.9.0-SNAPSHOT.pom was refused by the server: Unauthorized

Flattr this!

„Incomplete statements not supported yet, sorry, you’ll have to retype“ Error with Scala and Eclipse

Posted on 15. November 2013 Comments

I got this error defining the function

object test {
  val f:AnyVal => String = {
      case 10.0 => "10.0"
      case _: Double => "Double"
      case _: Int => "Int"
      case _: Byte => "Byte"
      case _ => "anything else"

 -> Incomplete statements not supported yet, sorry, you’ll have to retype.

However, in REPL it worked just fine.

repat@laptop:~$ scala
Welcome to Scala version 2.10.3 (OpenJDK 64-Bit Server VM, Java 1.6.0_27).
Type in expressions to have them evaluated.
Type :help for more information.

scala>   val f:AnyVal => String = {
     |       case 10.0 => "10.0"
     |       case _: Double => "Double"
     |       case _: Int => "Int"
     |       case _: Byte => "Byte"
     |       case _ => "anything else"
     |   }
f: AnyVal => String = 


So aparrently the error is in Eclipse. I use the version 3.7.2. Have you tried turning it off and on again? 😉

Flattr this!