Convert Docx to Markdown

I needed to convert a Docx file to Markdown, but Pandoc kept giving me this obnoxious error:

$ pandoc test.docx -o test.md
pandoc: Cannot decode byte '\xae': Data.Text.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

However, you can use the tool unoconv to make an intermediary step to convert first to HTML and then to Markdown.

$ unoconv --stdout -f html test.docx | pandoc -f html -t markdown -o test.md

On Ubuntu (And other Debian-based systems I would imagine)  you can get unoconv with a simple apt-get install unoconv.

Duplicate a Django modelformset_factory Form

I created a formset_factory and wanted to have a simple “click me to add another form”. This seemed like a routine task, but the solutions I found online were unnecessarily complicated or required me to install a separate Django app, which I had no intention of doing.

So I created my own…

The only pre-requirement that this needs besides standard Django is jQuery.

So here is a rough overview of how this works:

  • Create a modelformset in my views.py and send it to the template.
  • Add in a link that’s executed to trigger the new form adding.
  • Django’s formset_factory’s required management_form creates the id_form-TOTAL_FORMS hidden variable. The jQuery must update this value.
  • Have jQuery locate the current form and create a blank copy from.
  • Update the name and id parameters of the copied form using jQuery
  • Identify where to paste the new form
  • Paste it there!

Here is my views.py snippet. In my case, the model is called “Component” and the related form is called “ComponentForm”. I define them as follows:

ComponentFormSet = modelformset_factory(Component, form=ComponentForm)
componentformset = ComponentFormSet( queryset=Component.objects.none() )

The queryset must be set to Component.objects.none() for whatever reason, otherwise you will get the latest Component value. I pass the “componentformset” in the context to the template.

Next, the HTML must be rendered in the template as follows. Noticed that I used componentformset.0 with the “.0“. Why? Because componentformset, being a modelformset_factory, is a set of forms, not an individual form. We have an individual form, so I am only going to display the first one in the list. After that, the template includes a tbody called “formtemplate”. This will later tell our jQuery where to copy our template from. Finally, the <a> tag is necessary to tell jQuery when to add a new form. The newcomponents 

is where the new form will be pasted.

{{ componentformset.management_form }}
<tbody id="formtemplate">
{{ componentformset.0 }}
tbody>

<tbody id="newcomponents">
tbody>
<a href="#" id='addForm'>Add Componenta>

Next is the Javascript. (In my actual production code I put this above the HTML, but you could really put this anywhere since it’ll only trigger when the page fully renders).

<script type="text/javascript">
    $(document).ready(function() {
      var addForm = $('#addForm');
      var formNumberObject = $('#id_form-TOTAL_FORMS');

      var addForm = function(e) {
        e.preventDefault();
        newFormCount = parseInt(formNumberObject.val()) + 1;
        formNumberObject.val( newFormCount );

        var tabletemplate = $('#formtemplate tr').clone();
        var pastehere = $('#newcomponents');

        changevalues = tabletemplate.find('[id^=id_form-0]');
        changevalues.each( function(i, index) {
          currentid = $(this).attr('id').replace(/(id_form-)[0-9]+/, 'id_form-' + (newFormCount-1) );
          $(this).attr('id', currentid);

          currentname = $(this).attr('name').replace(/(form-)[0-9]+/, 'form-' + (newFormCount-1) );
          $(this).attr('name', currentname);

        });

        tabletemplate.appendTo(pastehere);
      }
      addForm.on('click', addForm);
    });
  
script>

The JS does the following. This is in order of the logic of the program, not in order of the code above, but you should be able to piece it together.

  • Identify the ‘addForm’ <a> link and set it to execute the function ‘addForm’ (yes, they have the same name, but you can change that)
  • Identify the id_form-TOTAL_FORMS by ID in the rendered HTML. Again, this is produced by the management_form part of our formset_factory.
  • Identify the number of forms. I suppose I could have just set this to 0 by default, but I’ll have the JS do it for me.
  • When the user clicks the ‘addForm’ link, it will execute the function ‘addForm’, which does the following:
    • Add 1 to the current newFormCount value.
    • Create a copy of the form and store it in “tabletemplate”.
    • Identify where to paste the data, identified by “newcomponents”.
    • Find each instance where the ID is in the RegEx pattern “id_form-0”
    • Iterate for each instance, and replace the number 0 with the current number. This will keep the individual component name intact, but update the count.
  • Append the newly created form to the “pastehere” tbody.

With that, you should be able to add a new form with the click of a button!

Thoughts? Comments? Please do let me know so I can fix any mistakes or update the code as needed. And remember…Free Palestine!

IPv6 Firewall Rules

I setup a Hurricane Electric tunnel to get my house on IPv6 (Verizon fails to deliver!) and was given a /64 allocation. I then setup a Router Advertisement daemon to get every computer online. Yippee!

But, there’s a problem…now every computer in my house is exposed to the wrath of the Internet. While the Network Discovery (ND) addresses are “random”, you can still intercept a client’s address through a variety of means. So I setup some basic IPv6 firewall rules to protect my clients.

Here is my script:

#!/bin/bash

# Default policy, this happens in the end
ip6tables -P FORWARD DROP

# Accept SSH
ip6tables -A FORWARD -p tcp --dport 22 -j ACCEPT

# Accept everything locally
ip6tables -A FORWARD -i eth0 -o he-ipv6 -j ACCEPT

# Accept all ICMPv6, kinda necessary 
ip6tables -A FORWARD -i he-ipv6 -o eth0 -p icmpv6 -j ACCEPT

# Accept all stateful connections, that we didn't initiate
ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

Here is an explanation:

  1. The default FORWARD policy is to drop all packets
  2. I accept all forwarded packets on port 22 (SSH) – This is because I frequently ssh into my personal machine while on the road.
  3. I accept all ICMPv6 packets: First because I want to be able to test pingings and such, but also because its required by IPv6
  4. I accept all packets for a connection that my clients initiated. This means I can arbitrarily connect out, but others cannot arbitrarily connect in.

I tested this out and it worked! There, now my personal home printer is not online 🙂

Giving credit where credit is due, borrowed a lot from Fabio Firmware.

Custom Django Fixture Imports

I needed to convert an XML document into a customize Django model with modifications to the based on programmable logic. Converting it to my model’s fixture would take too long and be unnecessary work, so I instead opted to manually convert the data.

I figured I could just import the Django model object, as is follows:

from tester.models import Control
a = Control()

However, I got the following vexing error in red:

$ python code.py
Traceback (most recent call last):
File "code.py", line 1, in
from tester.models import Control
File "/home/nahraf/src/beater/tester/models.py", line 5, in
class Control(models.Model):
File "/home/nahraf/src/beater/tester/models.py", line 6, in Control
family = models.CharField(max_length=40)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 1012, in __init__
super(CharField, self).__init__(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 146, in __init__
self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 46, in __getattr__
self._setup(name)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 40, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

In short, the solution is to set your Django application’s settings prior to importing the Django object. (My “tester” application is called “beater” cuz I beat up on it 🙂

My corrected code is as follows:

import os
# This must be executed before the import below
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "beater.settings")
import django
django.setup()
from tester.models import Control
Control()

After that, the code was able to import and utilize object. I hope this helps!

Free Palestine, Boycott Apartheid Israel!