I’d like to learn a lightweight scripting language that’s easy to use for data parsing / screenscraping. It would take text files of arbitrary format and turn them into CSVs and then perhaps put it into a SQL-like database on a remote server, or at least just upload the data to a remote API or SFTP server. The ultimate goal is to turn live data from embedded systems into visualizations on the web (but I expect the latter part to be handled by JavaScript/HTML5 visualization libraries).
The last time I tried this was ten years ago, using Perl, which I imagine has been superseded by less… inelegant… languages?
I started learning Lua a few weeks ago (for a game, initially), but it seems like the core library itself is very skinny and to do things like HTTP calls or CSV processing would require a lot of external libraries. I am not very familiar with the Lua “scene” and don’t know if these external libraries are stable and well-maintained, or if I’d have to deal with a lot of orphaned code.
A brief glimpse at Python, however, suggests that the standard feature set is already a lot more powerful: it’s got HTTP, CSV writers, proper regex, etc. all built-in.
Am I right that Python would probably be the better choice for this application? Or are there any other scripting languages that are better suited? I’d prefer to stay away from Java if possible, if only because the runtime is a huge pain to distribute and keep updated on Windows. And Java is not easy to learn.
I agree that Python is pretty much your sure bet if you’re looking for a scripting language. Lots of libraries to do lots of things. IMO it is the “standard” scripting language now.
Python would be a better choice than Lua. Lua (like JavaScript) is designed to be embedded in other applications, which is why it’s standard library is so scanty (also like JavaScript.) Python is designed to operate on its own, and therefore has a much more robust standard library.
It was Elder Scrolls Online, actually. As a new release, the addon scene was still young and exciting to get into WoW is way too mature for that.
Sadly, I very quickly ran into limitations in the ESO API, which was inconsistent, undocumented, and intentionally crippled. There was only so much I could do with it before I got bored, so I’m moving on to more interesting projects
Python is the standard now, eh? Is it often used online as well, replacing PHP?
My opinion: Python has pretty much usurped the role that Perl used to play. PHP has had more staying power for server-side close-to-presentation-layer code (I don’t think that python has the “embed bits of code right into your source” that people find useful, but I haven’t used it enough to be sure.)
Personally, I’m getting really into Scala and the associated frameworks, but if you don’t like Java, you won’t like that.
My personal favorite is C#, it has such pleasant tools and such a good debugger than while you might traditionally think of it as an “application” language, it’s great for scripting-type tasks like the one you describe as well.
PHP has the same issue for this task that Lua does: it was designed to do one specific thing, and you’d be using it for something else. Generally not a good idea.
I don’t think Python has any particular following in web applications. Generally, those are Java, C#, PHP, Ruby, Node.js. I know some data folks who love Python for scripting their data tasks, like grabbing data from APIs and inserting it into a database (similar to what you’d be doing.)
My own programming style is I spend 99% of my time in the debugger, so languages without good debuggers are out. (Meaning: Python, Ruby, PHP.) But everybody has their own preference.
I love .NET for Windows. Visual Studio is amazing. And beautiful.
But I never really understood how .NET works on other platforms. There is something called Mono, but it’s a small third-party project, not really supported by Microsoft, right? And it only works with a subset of .NET functions? How do I know if code I write on Windows will be portable right now, or three years from now?
MonoDevelop is more mature than any IDE I’ve ever seen for Python. Everything’s relative, of course.
Mono is supported by Microsoft in that .net is an open standard and Microsoft provides a reference implementation. It’s not like some sneaky hack or anything, but I also don’t think Microsoft actually funds them in any way. (Correct me if I’m wrong.)
My understanding is it implements all of .net 3.5 and most of .net 4. But again, I’m no expert in Mono.
Make sure you don’t use any Assemblies that start with “Microsoft.” The “System.” ones are all cross-platform, the “Microsoft.” ones are not, and apply only to Windows.
EDIT: I should add MonoDevelop runs fine in Windows, so if you do your development in MonoDevelop instead of Visual Studio you can guarantee the application will run cross-platform.
If you know Perl, Python is very easy to learn but, honestly, perl would still be great. Perl’s great strength is in manipulating text, there’s no other language that’s so optimized around that one purpose than perl. While it’s ugly and out of fashion nowadays, it’s still perfectly serviceable for munging around in the guts of a text file.
Thank you for this info! It’s really good to know about MonoDevelop. I will keep it in mind for my next project. This particular one is probably better suited to the grassroots FOSS languages, if only because I have a hard(er) time imagining the low-power embedded computing community embracing C#. There is .NET Micro and Netduino, but they’re not very popular, as far as I can tell.
In fact, after realizing that Sourceforge’s internal search engine really sucks, I Googled it instead and found a library (written in Python, of course) that provides two-way communication with my datalogger device. That’s 95% of the work already done. I learned some basic Python earlier today and will be experimenting more.
My god, there are SO MANY IDEs for Python. I thought Eclipse had become a standard IDE, but apparently not. There’s at least a dozen different free and commercial ones. It’s kinda overwhelming and I might just have to stick to my text editor
I loved Perl back in the day – the same way one might love a 3-pound Swiss Army Knife that only fits in the car trunk – but I can’t remember the last time I saw anything written in it. Ideally I’d like to contribute this and other projects back to the community, and it would be weird to deliberately choose a less-popular language, wouldn’t it?
In the few hours I spent learning Python, I’ve already started to appreciate its syntax. Even something as simple as {tabs-as-braces} makes typing a hell lot easier. Lua is somewhat “cleaner”, but Python seems vastly more powerful. I guess there’s always a tradeoff. Anyway, I don’t want to jump to any conclusions without knowing the languages in question. Might as well play with Python a bit to see how I like it
My fiancee maintains a single legacy script in perl at her work. It is the worst part of her job. The perl folks always bragged about how there are so many ways to do any task in perl, which is true, but it also meant that a maintainer can’t count on any consistency when attempting to figure out what the code does. If the code is not well-documented (and code never is) it really is a “write only language”.
Partially, that’s what python’s forced indentation is meant to avoid. It helps prevent people from writing code that’s hard to maintain.
BTW, you mention SourceForge, but have you used GitHub? GitHub is kind of the up-and-coming site for storing code you want to make public (or keep private, but you have to pay for private).
I code Python for a living, and I can recommend Pycharm - there is a free community edition that should do all you need.
Python will be ideal for what you want - it sounds a lot like projects some of my friends are involved in, that use Python - bioinformatics stuff, and the SKAdatastream.
