Category: Work

Up to now I’ve had to write fairly plain C++ due to having to support a wide variety of environments and compilers, but we’ve recently finally gotten everything in place to support at least C++17 in most of our projects, so hey, it’s time to learn what I can do now!

Structured Bindings

One of the big new features is structured bindings, which can split an object apart into separate variables. It’s not really all that useful on just plain objects, you may as well just reference them by member name then, but it really shines in a few specific scenarios, like iterating over containers like maps, where you can do:

// Old way:
for(auto& iter : someMap)
{
    if(iter.first == "something")
        doStuff(iter.second);
}

// New way
for(auto& [key, value] : someMap)
{
    if(key == "something")
        doStuff(value);
}

so you can give more meaningful names to the pair elements instead of ‘x.first’ and ‘x.second’, which has always irritated me. It’s a small thing, but anything that improves comprehensibility helps.

The other nice use is for handling multiple return values. Only having one return value from a function has always been a bit limiting; if you wanted a function to return multiple values, you had a few options with their own tradeoffs:

• Kludge it by passing in pointers or references as parameters and modify through them. This in turn meant that the caller had to pre-declare a variable for that parameter, which is also kind of ugly since it means either having an uninitialized variable or an unnecessary construction that’s going to get overwritten anyway.
• Return a struct containing multiple members. A viable option, but now you have to introduce a new type definition, which you may not want to do for a whole bunch of only-used-once types.
• Return a tuple. Also viable, but accessing tuple elements is kind of annoying since you have to either use std::get, or std::bind them to pre-existing variables, which runs into the pre-declared variable problem again.

Structured bindings help with the tuple case by breaking the tuple elements out into newly-declared variables, making them conveniently accessible by whatever name you want and avoiding the pre-declaration problem.

// old and busted:
bool someFunc(int& count, std::string& msg);

int count;       // uninitialized
std::string msg; // unnecessary empty construction
bool success = someFunc(count, msg);

// new hotness:
std::tuple<bool,int,std::string> someFunc();

auto [success, count, msg] = someFunc();

The downside is that the tuple elements are unnamed in the function prototype, which makes them a bit less self-documenting. If that’s important to you, returning a structure into structured bindings is also a viable option, where you can often use an IDE to see the order of members of the returned struct and their names.

Update: You could also return an unnamed struct defined within the function itself, so you don’t need to declare it separately. You can’t put the struct declaration in the function prototype, but you can work around this by returning ‘auto’. This should give you a multi-value return with names visible in an IDE. The downside is that you can’t really do this cross-module, since it needs to be able to deduce the return type. (Maybe if you put it in a C++20 module?)

auto someFunc()
{
    struct { bool success; int count; std::string msg; } result;
    ...
    return result;
}

...
auto [success, count, msg] = someFunc();

Optional Values

One of the scenarios I started using structured bindings for was the common situation where a function had to return both an indication of whether it succeeded or not (e.g., a success/failure bool, or integer error code), and the actual information if it succeeded. The trouble when you’re returning a tuple though is that you always have to return all values in the tuple, even if the function failed, so what do you do for the other elements when you don’t have anything to return? You could return an empty or default-constructed object, but that’s still unnecessary work and not all types necessarily have sensible empty or default constructions.

That’s when I discovered std::optional, which can represent both an object and the lack of an object within a single variable, much like how you might have used ‘NULL’ as an “I am not returning an object” indicator back in ye old manual memory allocation days. The presence or lack of an object can also represent success or failure, so now I find myself often returning an std::optional and checking .has_value() instead of separately returning a bool and an object when it’s a simple success/failure result. If the failure state is more complicated or I need to return multiple pieces of information, then structured bindings may still be preferable.

It’s also been useful where a rule or policy may be enabled or disabled, and the presence or lack thereof of the value can represent whether it’s enabled or not. (Though if it can be dynamically enabled or disabled then this might not be appropriate since it doesn’t retain the value when ‘unset’.)

struct OptionalPasswordRules
{
    std::optional<int> minLowercase;
    ...
};

if(rules.minLowercase.has_value())
{
    // Lowercase rule is enabled, check it
    if(countLower(password) < *rules.minLowercase)
       ...
}

Initializing-if

Another new feature that’s been really useful is the initializing-if, where you can declare and assign a variable and test a condition within the if statement, instead of having to do it separately.

if(std::optional<Foo> val = myFunc(); val.has_value())
{
    // We got a value from myFunc, do something with it
    ...
}
else
{
    // myFunc failed, now what
}

The advantage here is that the variable is scoped to the if and its blocks, avoiding the common problem of creating a variable that’s only going to be tested and used in an if statement and its blocks but that then lives on past that anyway.

Variant Types

This one is a bit more niche, but I’ve been doing a bunch of work with lists of name/value pairs where the values can be of mixed types, and std::variant makes it a lot easier to have containers of these mixed types. With stronger use of initializer lists and automatic type deduction, it’s even possible to do things like:

using VarType = std::variant<std::string,int,bool>;
std::string MakeJSON(const std::vector<std::pair<std::string, VarType>>& fields);

auto outStr = MakeJSON({ { "name", nameStr },
                         { "age", user.age },
                         { "admin", false } });

and have it deduce and preserve the appropriate string/integer/bool type in the container.

String Views

I’ve talked about strings before, and C++17 helps make things a bit more efficient with the std::string_view type. If you use this as a parameter type for accepting strings in functions that don’t alter or take ownership of the string, both std::string and C-style strings are automatically and efficiently converted to it, so you don’t need multiple overloads for both types. It’s inherently const and compact, so you can just pass it by value instead of having to do the usual const reference. And it can represent substrings of another string without having to create a whole new copy of the string.

bool CheckString(std::string_view str)
{
    // Many of the usual string operations are available
    if(str.length() > 100) ...
    // No allocation cost to these substr operations
    if(str.substr(0, 3) == "xy:")
        auto postPrefix = str.substr(3);
}

std::string foo("slfjsafsafdasddsad");
// A bit more awkward here since it has to build a string_view before
// doing the substr to avoid an allocation
CheckString(std::string_view(foo).substr(5, 3));
// Also works on literal strings
CheckString("xy:124.562,98.034");

The gotcha is that string_view objects cannot be assumed to be null-terminated like plain std::strings, so they’re not really usable in various situations where I really do need a C-compatible null-terminated string. Still, wherever possible I’m now trying to use string_view as the preferred parameter type for accepting strings.

A lot of this is fairly basic stuff that you’ll see in a million other tutorials around the net, but hey, typing all this out helps me internalize it…

One of our DLLs was lacking in logging, so I spent a bit of time adding a bunch of new logging calls, using variadic templates and boost::format to make the interface fairly flexible, much like in a previous post. However, I noticed that after doing so, the size of the DLL had increased from 80kB to around 200kB.

Now that’s not exactly going to break the bank, especially on newer systems where even a VM will probably have 4+ GB of RAM, but that kind of large change still kind of irks me. Modern languages let you do a lot more things a lot more easily, and 99% of the time it’s pointless to try and count every byte of RAM and instruction cycle you’ve spent, but I still kind of have a lurking fear that it might also let me get a bit too…sloppy? If I keep at it, will I eventually turn an application that runs fine on a 2GB system into one that needs 4 or 8 GB?

In this case, from the map file and some tweaking, I can break down this change into various parts:

• 50kB from code pulled in from Boost. Although I’m only directly using boost::format, that class is going to pull in numerous other Boost classes as well. At least this is generally a one-time ‘capital’ cost, and I can now use boost::format in other places without incurring this cost again.
• 24kB of templates generated from the logging interface. Since I’m using a recursive template approach to pick off and specialize certain types, a lot of different variants of the logging function get generated.
• 32kB for new calls to the logging interface. This is across 80 new calls that were added, so each new logging message is adding about 400 bytes of code. That seems like a lot just to make a function call, even accounting for the message text.
• 4kB in exception handling. Not a big deal.
• And 10 kB of miscellaneous/unaccounted for code. Also not going to worry about this too much. Rounding to pages makes these smaller values kind of uncertain anyway.

So, I guess the increase in size does make sense here, though I’m not sure if I can really do much about it. If I switch away from boost::format, I’d lose all its benefits and have to reimplement my own solution, which I certainly don’t want to have to do from scratch. sprintf-based solutions would have to be non-recursive, and that wouldn’t let me do the type specializations I want.

I might look at the assembly and see just where those 400 bytes per call are going, but that’ll probably only save a dozen or so kB at best for a lot of work. It may irk me, but in this case I’ll probably just have to live with it.

I’ve had some extra time to do a bit of code cleanup at work, and I decided a good refactoring of string usage was way overdue. Although this code has been C++ for a long time, the original authors were either allergic to the STL or at least uncomfortable with it and hardly ever used it anywhere; this was originally written a good 15-20 years ago, so maybe it was still just too ‘new’ to them.

In any case, it’s still riddled with old C-style strings. They aren’t inherently bad, but it does tend to lead to sloppy code like:

char tempMsg[256];
setMsgHeader(tempMsg);
sprintf(tempMsg + strlen(tempMsg),
        "blah blah (%s) blah...", foo);

That’s potentially unsafe, so it really should use snprintf instead:

char tempMsg[256];
setMsgHeader(tempMsg);
snprintf(tempMsg + strlen(tempMsg),
         sizeof(tempMsg) - strlen(tempMsg),
         "blah blah (%s) blah...", foo);

Ugh, that’s ugly. Even after making it safer, it’s still imposing limits and using awkward idioms prone to error (quick, is there a fencepost error in the strlen calculations?). So I’ve been trying to convert a lot of this code to use std::string instead, like:

std::string tempMsg = getMsgHeader();
tempMsg += strprintf("blah blah (%s) blah...", foo); 

which I feel is far clearer, and is also length-safe (strprintf is my own helper function which acts like sprintf but returns an std::string).

It’s usually fairly straightforward to take a C string and convert it to the appropriate C++ equivalent. The main difficulty in a codebase like this is that strings don’t exist in isolation. Strings interact with other strings, get passed to functions, and get returned by functions, so anytime you change a string, it sets off a chain reaction of other things that need to be changed or at least accommodated. The string you just changed to an std::string is also an output parameter of the function? Welp, now the calling function needs to be converted to use std::string. Going to use .c_str() to call a function that takes plain C strings? Oops, it takes just ‘char *’ and now you gotta make it const-correct…

To try and keep things manageable, I’ve come up with the following guidelines for my conversion:

Don’t try and convert every string at once; triage them.

A lot of strings will be short and set once, used, and then discarded, and they’re not really so much of a risk. Instead, focus on the more complex strings that are dynamically assembled, and those that are returned to a caller. These are the cases where you really need to be wary of length limits with C strings, and you’ll gain a lot of safety by switching to std::string.

Functions that take an existing ‘const char *’ string pointer and only work with that can be considered lower-priority. Since they’re not altering the string, they’re not going to be the source of any harm, and changing them could risk introducing regressions.

Use wrapper functions to ‘autoconvert’ strings.

If you have functions that already take ‘const char *’ strings as parameters and work perfectly fine, you don’t necessarily need to convert the whole thing right away, but it can help to have a wrapper function that takes an std::string and converts it, so that you can use std::strings in the callers.

void MyFunc(const char *str)
{
    ...
}

void MyFunc(const std::string& str)
{
    MyFunc(str.c_str());
}

Now functions that call MyFunc can be converted to use std::string internally and still call MyFunc without having to convert MyFunc as well, or having to pepper the caller with .c_str() calls.

This can get tricky if you have functions that take multiple string parameters and you still want to be able to mix C-style and std::string as parameters. If it’s only a couple of parameters you can just write all possible combinations as separate wrappers, but beyond that it gets unwieldy. In that case you could use template functions and overloading to convert each parameter.

void MyFunc(const char *foo1, const char *foo2, const char *foo3, const char *foo4)
{
    printf("%s %s %s %s\n", foo1, foo2, foo3, foo4);
}

const char *ToC(const char *x) { return x; }
const char *ToC(const std::string& x) { return x.c_str(); }

template<typename S1, typename S2, typename S3, typename S4>
void MyFunc(S1 foo1, S2 foo2, S3 foo3, S4 foo4)
{
    MyFunc(ToC(foo1), ToC(foo2), ToC(foo3), ToC(foo4));
}

int main()
{
    MyFunc("foo", "bar", "baz", "blah");
    MyFunc("aaa", std::string("fdsfsd"), "1234", "zzzz");
    MyFunc(std::string("111"), "222", std::string("333"), std::string("444"));
    return 0;
}

It’s still kind of awkward since it needs helper overload functions polluting the namespace (I’d prefer lambdas or something nested function-ish, but I can’t see a good way to do it that isn’t ugly and difficult to reuse), but it avoids having to write 2^N separate wrappers.

Don’t get too fancy while converting

std::string will let you be more flexible when it comes to string manipulation, and you’ll probably no longer need some temporary string buffers, some string assembly steps can be combined, sanity checks can be removed, other string operations can be done more efficiently in a different way, etc. But if you try and make these changes at the same time you’re converting the string type, you could risk messing up the logic in a non-obvious way.

Make the first pass at string conversion just a straight mechanical one-to-one conversion of equivalent functionality. Once that’s done and working, cleaning up and optimizing the logic can be done in a second pass, where it’ll be clearer if the changes maintain the intent of the code without being polluted by all the type change details as well.

There are some other caveats to watch out for, too:

Beware of temporaries.

Wherever you still need C-style strings, remember that a returned .c_str() pointer only remains valid as long as the std::string still exists and is unchanged, so beware of using it on temporary strings.

const char *s = (boost::format("%s: %s") % str1 % str2).str().c_str();
SomeFunc(s); // WRONG, 's' is no longer valid

However, the following would be safe, since the temporary string still exists until the whole statement is finished, including the function call:

SomeFunc((boost::format("%s: %s") % str1 % str2).str().c_str());

Of course, most people already knew this, but the odds of inadvertently introducing an error like this goes way up when you’re changing a whole bunch of strings at once, so it’s worth keeping in mind.

Variadic functions are annoying

Unfortunately, ye olde variadic functions don’t work well with std::string. It’s most often used for printf-style functions, but an std::string doesn’t automatically convert to a C-style string when passed as a variadic parameter, so you have to remember to use .c_str() whenever passing in an std::string.

There are non-variadic alternatives like boost::format that you might want to prefer over printf-style formatters, but if you’re stuck with printf-style functions for now, make sure you enable compiler warnings about mismatched printf parameters and set function attributes like GCC’s __attribute__ ((format …)) on your own printf-style functions.

If you have boost::format available, and a C++17 compiler, but don’t want to convert a bajillion printf-style parameter lists, you could use a wrapper like this with variadic templates and fold expressions:

template<class... Args>
std::string strprintf(const char *format, Args&&... args)
{
    return (boost::format(format) % ... % std::forward<Args>(args)).str();
}

With this you can then safely pass in either C-style or std::string values freely. Unfortunately then you can’t use printf-format-checking warnings, but boost::format will be able to do a lot more type deduction anyway.

Performance is a concern…or is it?

C-style strings have pretty minimal overhead, and with std::strings you’ve now got extra object memory overhead, reallocations, more temporary objects, generated template functions, etc. that might cause some performance loss. Sticking with C-style strings might be better if you’re concerned about performance.

But…with modern optimizers and move constructors, return value copy elision, small string optimizations in the runtime, etc., the performance penalty might not be as bad as you think. Unless these strings are known to be performance-critical, I think the safety and usability value of std::strings still outweighs any slight performance loss. In the end, only profiling can really tell you how bad it is and it’s up to you to decide.

(edit: Of course it wasn’t until after writing this that I discovered variadic macros, available in C++11…)

While working on some old code, there were a bunch of rather cumbersome printf-style tracing macros where you had to make sure the name of the macro matched the number of parameters you wanted to use. E.g., use TRACE1(“foo: %s”, s) for one format parameter, use TRACE2(“foo: %s %d”, s, x) for two parameters, etc. It was always annoying having to make sure you picked the right name, corrected it whenever the parameters changed, and so on.

I can understand why someone created these macros in the first place. They exist as a shorthand for another variadic tracing function with a much longer name, to keep tracing calls short and snappy, or to redirect them to nothing if you want tracing disabled, but that presents a few challenges. An inline function would preserve the preprocessor-controlled redirection, but you can’t simply have a function with a shorter name call the other function because variadic functions can’t pass on their arguments to another variadic function. You could just create a #define to map the short name to the longer name, like “#define TRACE GlobalTraceObj::CustomTraceFunc”, but that risks causing side effects in other places that use ‘TRACE’ as a token, and doesn’t let you eliminate the call if desired. A parameterized macro avoids that problem, but only works for a specific number of parameters, and hence you wind up needing a different macro for each possible number of arguments.

I figured there had to be a better way and hey, now that I have newer compilers available, why not try C++11 variadic templates? They do let you pass variadic arguments on to another variadic function, which is exactly what’s needed, and it can still be conditional on whether tracing is enabled or not!

template<typename... Args>
inline void TRACE(const char *format, Args... args)
{
#ifndef NOTRACE
    GlobalTraceObj::CustomTraceFunc(format, args...);
#endif
}

And it worked perfectly, now I could just use TRACE(…) regardless of the number of parameters.

Except, I got greedy… Another nice thing is to have the compiler do consistency checks on the printf-style formatting string and its arguments, which you can do with compilers like GCC with a function attribute like on the main tracing function:

static void CustomTraceFunc(const char *format, ...)
#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif

I wanted that same checking on the TRACE wrapper function, but it turns out that at least at the moment, you can’t apply that same function attribute against a variadic template function; GCC just doesn’t recognize the template arguments as the right kind of variadic parameters that this attribute works on. Oh well.

I really wanted that consistency checking though, so in the end I abandoned the variadic template approach and just wrote TRACE as a plain old variadic function, which meant having to modify the other tracing functions to use va_lists instead, but that wasn’t too big a deal. If I didn’t also have control over those other functions, I would have been stuck again.

#ifdef __GNUC__
    __attribute__ ((format (printf, 1, 2)))
#endif 
inline void TRACE(const char *format, ...) {
#ifndef NOTRACE
    va_list args;
    va_start(args, format);
    GlobalTraceObj::CustomTraceFuncV(format, args);
    va_end(args);
#endif
}

static void GlobalTraceObj::CustomTraceFuncV(const char *format, va_list args)
...

For some reason I got it into my head last weekend to set up a Linux s390x instance, using the Hercules emulator. We do some mainframe stuff at work, and we have a couple instances of z/Linux already, but they’re in Germany and managed by another team, so maybe it would be neat to have our own local instance, even if emulated? Although I’ve done some work on the mainframe before (mainly on the USS side), I’m hardly an expert on it, but how hard could it be?

So fine, I set up a Fedora Server 30 VM as the emulator host and installed Hercules onto it and set up a basic configuration, per various guides. There are a handful of s390x Linux distros but I figured that Debian would make for a nice, generic baseline instance, so I grabbed the Debian 9.9 s390x install DVD image.

Problem #1: It wouldn’t boot. IPLing from the DVD image just spit out a few lines of output and then hung. After some digging, this had been noted on the Debian bug mailing list, but with no resolution or workaround.

Figuring that maybe the distro was too new for the emulator, I grabbed a Debian 7 install DVD (there was no listing for an s390x version of 8 on the main download pages) and hey, it actually booted and started going through the install process.

Problem #2: It doesn’t actually install from the DVD. Even though it has all of the packages on it, the s390x installer still goes and gets them from the network, and the networking wasn’t working. It could ping out to other networks, but DNS and HTTP wouldn’t work. After way too much fiddling around, I finally figured out it was an iptables ordering problem, and using ‘iptables -I …’ instead of ‘iptables -A …’ on the forwarding commands worked around that and got networking going.

Problem #3: The mirrors didn’t have Debian 7. Unfortunately I didn’t realize beforehand that the Debian 7 packages were no longer available on the mirror sites, so the installer couldn’t proceed. With a bit of searching, I found that there actually was a Debian 8 disc for s390x though, so I got that and gave it a try.

Problem #4: The mirrors don’t really have Debian 8, either. At least not the s390x packages, just the other main platforms. At this point it looked like there just wasn’t a path to get any version of Debian working, so I started trying some of the other distros.

Problem #5: The other distros aren’t quite as easy to install. Newer Fedora releases hung, older Fedora releases didn’t quite behave as expected and it was unclear how to make installation proceed, and openSUSE was still experimental and unclear how to install it. I even tried Gentoo, which seemed to work for a while after starting up before hanging at a point where it was unclear if it was grinding away at something intensive or not, and I let it sit there for two days before giving up on it. So yeah, not much luck with the other distros either.

Searching around for more info, I found that there were some newer versions and forks of Hercules that potentially fixed the hang problem, so it was time to give Debian 9.9 another try, using the Hyperion fork of Hercules.

Problem #6: Hyperion’s still a bit buggy. It compiled and installed just fine, but some of the permissions seemed incorrect and I had to run it as root. Even before IPLing it was extremely sluggish (sending the load average up to over 8), and trying to IPL the Debian disc just froze in an even earlier spot. So much for that.

Then I gave the ‘spinhawk’ fork of Hercules a try, and…hallelujah, everything’s gone smoothly since. It IPLed from the Debian image fine, it could find the mirrors and download packages, partition the disk, etc., and I now have a fully installed and working s390x Linux system.

Was it worth the hassle? Eh, probably not, I’m still better off doing any coding for work on our actual non-emulated z/Linux systems. It was interesting just to experiment and play around with for a bit, though.

I had an odd problem after rebooting one of my work computers yesterday. When I went to log in, it said that it couldn’t load the local profile, that it could be lost or corrupt, and created a temporary profile for me to log into instead.

All of my usual files were there in my profile directory, so it wasn’t because I’d lost some critical files. I rebooted a couple of times in case it was a temporary thing, but it continued to happen. I also booted into safe mode and could log in just fine there, so the files weren’t corrupt either. After poking around the logs a bit, the cause of the error seemed to be that the profile files were in use when it went to try and load them. A bit of googling revealed that this is a common problem when using migrated profiles, but that wasn’t the case here.

But, then I realized that a virus scan was being kicked off when the system booted, since one hadn’t been run in a while, and one of the first sets of directories it scans is…the user profiles. Symantec AV was probably scanning my profile directories at the time I was trying to log in and causing sharing conflicts on the files.

After rebooting again, I waited five minutes before trying to log in, and everything worked fine once more.

So, if I’m not really happy with IDEs, what can I do to improve my development environment? Well, I’ve discovered a few other tricks…
Continue reading “Dis-Integration”

My development environment at work is pretty primitive, being a collection of largely unrelated programs loosely tied together with some scripting. Integrated environments are where it’s supposedly at nowadays though, so I figured I should give some of them a shot.
Continue reading “Integrate Me”

I just spent over an hour struggling to get a set of class templates to even compile properly. It kept complaining that a base class was undefined, so I kept tinkering with the template parameters, thinking that I was using them incorrectly or there was some subtle behaviour that I was failing to account for, since it’s a moderately complex template (multiple levels of derivation, multiple template parameters at each level).

But no. It was simply because I had copied the base class header file for the derived classes and forgot to change the multi-inclusion-preventer #ifdef on one of them, making it completely skip over the definition of one of the intermediate classes. Duh…