Boot without starting X-server
With the Raspbian image, you can re-run the initial start up script using:
$ sudo raspi-config
and entering your sudo password.
This will bring up the same menu options that you got after first boot.
You do not need to remake all your first boot choices, just use the arrow keys to move to the menu options you want to change.
In your case, selecting:
Start X-server after boot?
no or disable
will sort you out.
Editing to add non gui based options:
As noted in the comments below by @mirk, the exact command for the raspberry is:
sudo update-rc.d lightdm disable
The raspi-config script creates a GUI menu which allows you to make several system config changes. This portion of the script enables or disables lightdm, the desktop manager used for Raspbian. It uses the same update-rc.d command as is described by Bryan Dunmore. Except Raspbian does not use GDM by default, and the RasPi will barely run it if installed.
That's not the answer to the question imho. With raspi-config you setup to boot to CLI permanently while the question asks how to do it once.
How X.Org works is that you have a desktop manager that boots the X Server for you. That means that in order to prevent X Server from starting, you need to remove your destkop manager from the daemons list.
If you are using GNOME then you're desktop manager is GDM. If you're using KDE you're desktop manager will be KDM. For this, I'm going to assume that you're using GNOME; ergo you are using GDM as your desktop manager.
In Raspian, the desktop manager is called LightDM.
To disable a daemon in Debian (which Raspbian is a fork of) you have to run the following command.
$ sudo update-rc.d <daemon> disable
So in order to disable LightDM we have to run the following command.
$ sudo update-rc.d lightdm disable
And now on your next reboot LightDM shouldn't start and will, in turn, not start X Server.