Freebies in SA (or How to download IP address lists to a Mikrotik)

In the state of Australia I currently live in, there are 2 major ISP’s – Internode and Adam Internet.
Being a typical nerd and living with other typical nerds we of course have 2 ADSL connections, 1 to each of these providers which are shared out amongst the house.

Internet Cafe. Now where's my Mocha?

However each of these providers have what we call in Australia – “unmetered content” that is to say download traffic from these sources is not measured and removed from our download allowances (yes, we have limits on what we can download).

So anyway, I’ll make like a sharp stick and get to the point.

We wanted to make sure that free traffic from each ISP would always go via that ISP’s link to save us on downloads and each ISP offers a list of the free IP’s in a nice text format:

Internode – https://customer-webtools-api.internode.on.net/unmetered_ip_address_list.txt
Adam Internet – http://www.adam.com.au/unmetered/unmetered_ip_address_list.txt

Note: The Internode list is on a https only server and the mikrotik can only do a http fetch. So in my case, I’ve used a php file with a simple include function hosted on my own server and when I try to download that, it simply relays the latest copy of the Internode list.

Here’s the adam version of the scripts commented:
===
Script: fetch-adam-ips-list
===

#Fetch (download) adam address list from their webserver
#Log completion
#Delay for 4 seconds (in case download process gets stuck)
#Run the Address parsing script

/tool fetch address=www.adam.com.au host=www.adam.com.au mode=http src-path=unmetered/unmetered_ip_address_list.txt
:log info "Download Complete"
:delay 4
/system script run Add-Adam-Unmetered-IPs

===
Script: Add-Adam-Unmetered-IPs
===

#Log beginning of modification
#If the unmetered address list file size is larger than 0, begin
#Remove all previous entries
#Put the content of the txt file into the variable $content
#Get the total length of $content

:log info "Beginning Adam freee Address List Modification"
:if ( [/file get [/file find name=unmetered_ip_address_list.txt] size] > 0 ) do={
  /ip firewall address-list remove [/ip firewall address-list find list=Adam-Unmetered]
  :local content [/file get [/file find name=unmetered_ip_address_list.txt] contents] ;
  :local contentLen [ :len $content ] ;

#Set lineEnd to 0
#Set line to null
#Set lastEnd to 0

:local lineEnd 0
:local line ""
:local lastEnd 0  

#Beginning of do/while loop
#Set lineEnd to the first point in the line there's a newline char
#Set line to be whatever is between lastEnd and lineEnd
#Set lastEnd to be lineEnd + 1 (ready for new line)

  :do {
    :set lineEnd [:find $content "\n" $lastEnd ] ;
    :set line [:pick $content $lastEnd $lineEnd] ;
    :set lastEnd ( $lineEnd + 1 ) ;

#if the line is not empty (1st char is a newline) do the following
#if line is longer than 0 characters - add a new ip firewall address list entry with $line as the address
#End if segment
#End do segment - while $lineEnd is smaller than the total length of the list
#End if segment
#Log completion

    :if ( [:pick $line 0 1] != "\n" ) do={
      :if ( [:len $line ] > 0 ) do={ /ip firewall address-list add list=Adam-Unmetered address=$line }
    }
  } while ($lineEnd < $contentLen)
}
:log info "Adam free IP List Modification Complete"

You can then use this address list in your mangle rules, or alternatively modify this script to make it add static router etc.

The IP address list segment is very versatile and can be modified to suit a number of different cases
Previously I’ve used modified versions of this to:
a) scrape spam blacklists
b) add conficker domains to a blocklist
c) pull admin IP addresses from a server to add to an allow list.

Have fun and post back if you have another use for these!

Advertisement

8 thoughts on “Freebies in SA (or How to download IP address lists to a Mikrotik)

  1. Pingback: Anonymous
  2. Please write a script to syncronize address lists between mikrotik routers without need an external server

  3. Just convert the script to turn the addresses into marked routes then distribute these via BGP.
    This will allow you to synchronise easily between many routers while just 1 routers does the lookups.

  4. Hi,

    The script breaks when you have an IP list longer than 220 lines.

    Any ideas?

  5. Actually,

    I just tested again after 251 lines or contentLen becomes equal to 0. It works with up to 251 lines or contentLen = 4092.

  6. Correct, at this point in time MikroTik variable length limits are 4095 bytes because this is all that can be stored in the MikroTik provided text variable at this time.
    The introduction of LUA was intended to fix this (v4 beta) but was pulled before the final release due to issues.

    You can work around it somewhat by breaking up your variables however if you intend to read a file larger than that size you’re out of luck unfortunately. 🙁
    It’s worthwhile sending in a ticket and requesting re-introduction of LUA so they know people are still interested in it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.