I said applications programming, not embedded controllers. If your definition is that software engineering is done for embedded controllers then that is fine, but that is not the definition I use or that many others use.
java.util.ArrayList
Hm. I started writing a long diatribe to address some of Cooper’s comments in a general sense, and the post got quite long and rambly… not sure whether I’m making a valid point or how it would be received or if its even applicable to the OP…
Not sure if I should post it, if this topic has fallen off into space.
Post it. If nothing else, it’ll be a useful reference.
This bit I agree with. Software ‘Engineering’ (architecture, design, whatever your baileywick) isn’t just for the microcontrollers of the world. I’ve done development and design across a wide spectrum of languages and environments – in the end, you should still be bringing to bear your entire experience, because you often find that a microcontroller ‘trick’ in a lower level language has a useful analog for that higher level GUI problem you’ve been wrestling with (or visa versa).
It does not necessarily matter if you’re processing high rate DSP data in a microcontroller, or if you’re processing a large database in a high end VB application connecting to an SQL database. Some similar techniques may apply. If an SQL database function cannot be found to do a data analysis in the way you want, you may end up having to pull records into memory into data structures you need to operate on. If the operation you’re performing on the DSP data needs to be analysed in some sort of comparitive way, you may be doing the exact same data structures and operations.
Perhaps your tool is a tiny pair of tweezers or a huge all-16’ths crescent wrench, but it may still be a bolt you’re tightening.
There are analogues to problems in a high level and a low level domain no matter how elaborate your ‘standard’ tools are. At some point you may be forced to stray outside the boundaries of what your tools can give you, so Software Engineering applies at every level.
To extend my comment from the previous paragraph, I have to disagree with this (and its chain of comments in previous messages).
(this starts getting a little philosophical about what a software developer architect engineer type person should know… don’t know why I’m posting it here, except that it seems in the back of my mind to have some small relevance)
It will be precisely the difference between those who just use the tools blindly and those who know how the tools work and can fix or adapt them that will, in my opinion, ultimately be a dividing line in software development (maybe even drawing the line between a software engineer and a computer programmer). This partly comes from experience, but also from a desire to know and to make it better or use it appropriately.
Just knowing that the quicksort algorithm or a hash table has been provided as a prewritten tool is, to my mind, insufficient for mastery of these tools and for correct application of them to solve a problem. Quicksort, for example, depending on its implementation, may sort very poorly on an already sorted array. It is certainly not a guaranteed n log n sorting process, possibly degenerating to n^2 operation in some cases. Mergesort on the other hand is guaranteed n log n operation, provided you can acquire a block of memory the same size as your data set in better than n log n time.
Hash tables are only as good as your hashing function and specific to your data set, so you have to be more knowledgeable than ‘there’s this prewritten tool’ to use them effectively. On top of that, there are several flavours of hash tables who’s effectiveness are dependant again on your data set. So it requires some experience and knowledge of internal details to make good use of these tools.
Perhaps it is as a result of software ‘engineering’ being in its infancy that a quicksort is not a quicksort is not a quicksort and that you have to know some internal details of your building blocks, but I suspect civil and mechanical Engineers building a bridge know at least a little bit about the materials and machinery they’re using – more than just that they are there as tools to be used.
I’m not suggesting reuse of tools shouldn’t be practiced – of course it should, where reasonably safe and cost effective. However, saying that you don’t need to know how they work seems to me to imply that one should employ quicksort whenever one needs something sorted, disregarding any impliciations of using that particular implementation or that particular algorithm.
Furthermore, by knowing how these things work, a software developer is better equipped to adapt old solutions to new, weird, and wonderful problems.
Sometimes it doesn’t matter. Sometimes it does. Knowing when, and using that judgment to build a better bridge is to me a telling dividing line.
Case in point: In solving a problem, it became clear that I needed an extensible searchable collection of data without predefined upper bounds on the number of elements, and that searching would be a primary problem, and that data would be added fairly often. I had the requirement, though, to know how many repeats there were of an item, and how many were less than or equal to a given item, and I needed to know that information in particular very very quickly. At worst, I couldn’t allow a single one of these operations to require O(n) time.
There are lots of container tools/classes out there. I wanted to use one if I could. For my searching purposes, an extensible array or a binary search tree might be sufficient. However, for adding elements, the array was insufficient as I might have to insert an item in the middle. A linked list enabled a simple way to inject things in the middle of my data set, but searching became a problem. The binary search tree was a compromise. Even then, I had to make sure that the standard implementation of the BST kept the tree balanced to keep it from degenerating to a linked list.
No problem. There’s lots of binary search tree implementations out there even with the balancing requirement. There’s even some very well written library routines in most languages, in particular the one I was using, that enabled one to build a BST using a key for sorting and associated data going along for the ride in each node. The adding is sufficiently quick, and the searching is sufficiently quick.
All this came from academic understanding of what I wanted out of a BST, and provided the prewritten libraries live up to certain assumptions or specifications about behaviour I should be fine, right?
Except none of them provided a simple way to keep track of the number of repeats of an item quickly, because they added new items as new nodes despite the duplicate key values, and none of them provided a simple way to keep track of the number of items with a key value less than a given one short of searching a good portion of the data set and counting. I could do my own duplicate-counting by way of just a counter in the augmented data part of each node and doing a search before every add. However, keeping track of the number less than an item is best served by altering the code for the BST itself during an Add operation to maintain information on the size of left and right subtrees. At worst, if I had access to the tree (which I didn’t, directly, given that it was hidden behind the abstract interface) I could count the subtree size myself.
(At this point, I note to anyone reading this that I’m sure there could be countless replies to how I could have handled this, but I’m relating my experience for one reason or another)
So, either I alter the existing BST code, or I roll my own BST with the requisite properties or I could search for an existing implementation that had the requirements. However, using the default built in libraries is insufficient to handle my problem. Perhaps the BST tool is written sufficiently cleverly so that I can modify the Add operation via inheritance and polymorphism to solve my problem. Maybe. If I can live with this overhead.
No matter, though – it still requires that I know some fundamental properties about the data structure and how a BST works at the implementation level. Naturally I solved the problem, but the basic tools were insufficient to the task.
I think my point in all this is that it kinda got up my nose to see someone claiming that ‘I can use it, but I don’t really need to know how it works.’ To me, this is patently untrue when you need to adapt strange circumstances or variations on the basic theme or need to optimize some code based on some assumptions.
Standard tools do not solve all problems, and determining the appropriate applicability of a tool to the problem sometimes requires more than a cheat sheet on the tool’s usage. Sometimes it requires knowing many internal details.
There’s another take on this too… so what if you have a whole pile of predefined data types and tools… so now, do you decide to use a list of lists, or a vector of binary search trees? I guarantee you that no set of library tools will provide every combination of tools to solve a problem you’re after, and you still have to understand enough about the tools to make judgment calls on their combination.
Don’t mean to seem like I’m picking on Cooper. 
I’m rambling in a more general sense spurned by what might have been simply a passing comment. I use or adapt existing tools whenever I can find a way to do so. I’m not so far gone that I recommend rolling your own except when necessary.
But to again tie this back to the whole Software Engineering thing, I think these sort of experiences and judgment calls and further the requirement to actually make those judgment calls by the projects at hand and to be responsible for them is what separates the straight coders from the Engineers. Anybody can code up a solution involving data structure A or processing technique B given sufficient specs, but the decision on whether or not it is wise for each particular circumstance is to me the real issue.
This is an extremely small and specific example that hardly stresses the sorts of things a Software Engineer might be responsible for deciding, and not as grandiose as deciding which alloys are best suited to a particular bridge or which concrete and rebar combination will best survive the earthquake, but solving a problem correctly is to me more the correct application of experience than the direct usage of a pre-existing tool, even if the pre-existing tool is, in the end, used.
Er. as I mentioned earlier, I don’t know if this message has a place in this area…
William everything you said is correct - and I didn’t mean to say that I shouldn’t need to know how to determine which tool to apply to which job. I will point out that because most business programming is solving a narrow domain of problems (how to track X information, make sure I can relate it to Y, provide views and reports with Z latency) the standard tools will just about always solve the problem well enough. The statement I made is invariably based upon my own experience, which since college has been limited to business applications programming. Perhaps Software Engineering has little place there - almost all of our most import issues have been solved by the excellent Software Engineers who sell us applications servers and database systems.
If it were otherwise, most of what we do simply would not be affordable.
If we were to accept that “Software Engineer” applied to someone who could apply a wide range of computer science principals to both abstract and concrete problems, then what do we call it when we manage requirements, coding, testing and configuration management in a responsible, repeatable and optimized process? This latter part is important to any software project, whereas the former may not be.
Oy. You’ve got me there. Perhaps at that level, a Software Engineering Manager? After all, around my workplace, the Engineering Managers are responsible for organizing the various aspects of the individual processes being handled by those strictly allowed to call themselves Engineers.
I see what you’re saying, and I don’t have a terribly useful answer to this one. To some degree I have had to come up with and justify requirements, done testing, and had to deal with configuration management myself, but perhaps not as much as you yourself do (for me, testing most of all).
I’d still claim that in business applications ‘engineering’ wisdom might come from knowing which tool is most appropriate and to be able to justify why and be responsible for that decision.
Sigh, but aside from possibly “Software Engineering Manager” I really can’t answer your question adequately. Just goes to show what a tangled ugly pickle this is. Even my engineering comrades nearby with whom I chatted over lunch agree its not an easily resolved set of definitions and/or responsibilities, legal or otherwise.