A couple months ago I added profiling output to our build system (a fork of JamPlus). It’s a simple Chrome Tracing view, and adding support for that to Jam was fairly easy. Jam being written in C, I found a simple C library to do it (minitrace) and with a couple compilation fixes it was working and had profiler blocks around whatever tasks the build system was executing.

That by itself is not remarkable. However… once you have profiling output, you can start to…

Notice Things

The other day I was doing something unrelated, and for some reason looked at a profiler output of Unity editor build I’ve just done. “Common sense” about C++ codebases tells us that usually it’s the linking time that’s The Worst, however here it was not that:

There’s a whole gap right before linking starts, when everything else is done, just the build is waiting for one C++ file to compile. I was busy with something else right then, so added a TODO card to our team task board, to be looked at later.

Sometime later I was doing a different build (of just the “engine” without the editor bits), and looked at the profiler too:

Now that looks pretty bad. The total build time (this was a “full rebuild”) was ten minutes, and almost five of them were “wait for that one C++ file to finish compiling”. The file in question had a compile time of seven minutes.

This sounded like a build inefficiency that is too large to ignore, so I dug in for a bit.

Aside: at the moment our small “build team” has not even started to look into “full build” performance, for various reasons. Mostly repaying tech debt (with interest!), optimizing incremental build performance and improving things like IDE projects. Optimizing full build time we will get to, but haven’t just yet.

7 minutes compile time for one file? Craaazyy!

Yes, it is crazy. Average compile time for C++ files in this project and this config (win64 Release) is about 2 seconds. 400+ seconds is definitely an outlier (though there’s a handful of other files that take 30+ seconds to compile). What is going on?!

I did some experiments and figured out that:

• Our Lump/Batch/Unity builds are not the culprit; there’s one .cpp file in that particular lump that takes all the time to compile.
• MSVC compiler exhibits this slow compilation behavior; clang compile times on that file are way better (10+ times faster).
• Only Release builds are this slow (or more specifically, builds where inlining is on).
• The fairly old VS2010 compiler that we use is not the culprit; compiling with VS2015 is actually a tiny bit slower :(
• In general the files that have 30+ second compile time in Release build with MSVC compiler tend to be heavy users of our “SIMD math library”, which provides a very nice HLSL-like syntax for math code (including swizzles & stuff). However the implementation of that is… shall we say… quite template & macro heavy.
• That 7 minute compile file had a big SIMD math heavy function, that was templated to expand into a number of instantiations, to get rid of runtime per-item branches. This is in CPU performance sensitive code, so that approach does make sense.

Whether the design of our math library is a good trade-off or not, is a story for another day. Looks like it has some good things (convenient HLSL-like usage, good SIMD codegen), and some not so good things (crazy complex implementation, crazy compile times on MSVC at least). Worth discussing and doing something about it in the future!

Meanwhile… could some lo-tech changes be made, to speed up the builds?

Speeding it up

One easy change that is entirely in “build systems” area even, is to stop including these “slow to compile” files into lump/batch/unity builds. Lump builds primarily save time on invoking the compiler executable, and doing preprocessing of typically similar/common included code. However, all that is sub-second times; if a single source file takes 10+ seconds to compile then there’s little point in lumping. By itself this does not gain much, however for incremental builds people won’t have to wait too long if they are working on a fast-to-build file that would have ended in the same lump as a slow file.

Could also somehow make the build system schedule slow files for compilation as soon as possible. The earlier they start, the sooner they will finish. Ideally the build system would perhaps have historical data of past compile times, and use that to guide build task scheduling. We don’t have that in our build system (yet?)… However in our current build code, moving slow files out of lumps makes them be built earlier as a by-product. Good!

The above things didn’t actually speed up compilation of that 7-minute file, but were trivial to do and gained a minute or so out of full build time (which was 10 minutes originally).

And then I tried something that I didn’t have high hopes of working - in that slow file, factor out parts of the big templated function into smaller, less-templated functions.

Fairly trivial stuff; if you’re into fancy refactoring IDEs that is done by “select some code lines, press Extract Method/Function button”. Except this is C++, and the IDEs don’t quite get this right (at least in this code), so you do it manually.

I split off five or so functions like that and… compile time of that file went from 420 seconds to 70 seconds. That is six times faster!

Of course, factoring out functions can mean they are no longer inlined, and the original code will have to call them. So that’s time spent in passing the arguments, doing the jump etc. However, it could make parent function use registers better (or worse…), could result in lower amount of total code (less I$ misses), etc. We have profiled the code change on several platforms, and the difference in performance seems to be negligible. So in this particular case, it’s a go!

Now of course, a minute to compile a C++ file is still crazy, but additional speedups probably involve changing the design of our math library or somesuch – needs more work than someone clueless randomly trying to speed it up :)

The build profile looks much better after the change. No longer gaps of “all the CPU cores wait for one job to finish” type of stuff! Well ok, linking is serial but that is old news. Full build time went from 10 minutes down to 5min 10sec, or almost 2x faster!

Takeaways

• Add some sort of profiling data capture to everything you care about. Had I not added the profiler output, who knows when (or if?) someone would have noticed that we have one super-slow to compile C++ source file. In my case, adding this simple profiling capture to our build system took about a day… most of that spent learning about Jam’s codebase (my first change in it).
  • Seriously, just do it. It’s easy, fun and worth it!
• Every template instantiation creates additional work for the compiler’s optimizer; pretty much as if the instantiated functions were literally copy-pasted. If a templated function was slow to compile and you instantiate the template 20 times… well now it’s very slow to compile.
  • Factoring out parts into less-templated functions might help things.
• Complicated template-heavy libraries can be slow to compile (“well duh”, right). But not necessarily due to parsing (“it’s all header files”); the optimizer can spend ages in them too. In MSVC case in this code, from what I can tell it ends up spending all the time deciding what & how to inline.
• Speeding up builds is good work, or at least I like to think so myself :) Less time spent by developers waiting for them; better throughput and lower latency in build farms, etc. etc.

This June we’ve spent two weeks traveling in South Korea, so here’s a writeup and a bunch of photos.

Caveats: my first trip there, and very likely I mis-planned something or missed some obvious “oh my god can’t miss that” things to see or do.

Our trip was two weeks; with myself, my wife and two kids (14 and 8yo). If you’re going alone, or for a honeymoon, or with a group of friends, the experience might be very different.

Planning

We did the itinerary ourselves, using the same old method of wikipedia + wikitravel + random blogs + asking around. Planning via writing rough outline in google docs and marking up places in a custom google map.

And here was the first challenge: Google does not have that much information on Korea, especially in the maps department. Apparently this is due to an old law that prevents mapping/GIS data to be exported for foreign countries & companies (see this quora answer). Does not make much sense these days, but oh well.

It’s not a big problem, but made me realize how much I have learned to depend on Google for planning trips. “A lot”, and that’s slightly scary.

What we came up with was this: in two weeks, we go to Seoul (3 nights), Sokcho (2 nights), Gyeongju (1 night), Busan (3 nights), Jeju (2 nights), Seoul (2 nights). Did not have any concrete goals, except “see some cities, nature, people, temples, food”. Wanted to check out some K-pop concert, but about the only one happening during that time (of Monsta-X) got sold out in like two minutes :|

Getting Around

We did not rent a car, instead were traveling mostly by metro & bus. T-money card for local transit works conveniently; metro goes often & on time. Both Seoul and Busan and really large cities, so getting from one place to another takes a lot of time. Outside of big cities, figuring out how exactly to get from A to B, is a bit of a challenge. For example bus terminal in Sokcho barely has any information in English, and not many people speak it either.

As mentioned before, Google Maps does not have exhaustive information, so I used a bunch of other apps to figure out buses, trains, places to go and so on. Some of them are Korean only, but after some fiddling I was able to semi-randomly get by.

Speaking of language… my older kid has learned to read & write Hangul (and some Korean words) as a hobby, and that was useful! The writing looks intimidating for someone who sees it the first time, but actually it’s fairly simple! There’s only a few dozen letters; they are just arranged in a non-linear fashion (instead each syllable is arranged into a square block).

I do recommend to check it out. If nothing else, know that ㅇ at top is silent (does not do anything), and at the bottom is “ng”. With just that you can already tell some words apart!

One place where we should have rented a car is Jeju. It seems small, but public transport there is… not terribly good, shall I say. Rare, slow and crowded buses, and fairly long walking distances from bus stops to anything. Do rent a car or hire a fulltime taxi!

Staying

Wow, Korean apartments are small! It makes sense since South Korea is one of the most densely populated countries. We mostly used airbnb, and learned the hard way that many listings with “two rooms” are actually one area with two beds tucked to the ceiling and a narrow staircase to get into them :)

Stayed a couple nights in a hanok in Bukchon, and that was nice!

Aside about airbnb: lately overall it feels kinda “meh”. A number of years ago airbnb meant you could experience places that are somehow personal; places where someone lives. These days, more and more airbnb apartments are rented full-time, with the same darn Ikea stuff in all of them. Often you don’t even meet anyone, just get a message with they keypad code and that’s it. It’s kinda like going to a hotel, without any of the advantages of a hotel. Meh!

Culture

We learned that selfies are important, especially if you’re showing V sign or some sort of heart symbol in them. Overall they are pretty serious about that stuff:

So we tried to blend in,

Cult of “youthful, perfect” beauty is fairly big here, it seems. At least it’s kinda equally divided among sexes. Maybe even larger part of ads & posters with “here’s someone beautiful” image had a guy in them. Makeup is important, clothes are important, etc. I’ve read somewhere that “if you go with just t-shirt and jeans, people will think you’ve given up on life”. We went mostly in t-shirts and jeans :)

Feels like Koreans love to write up instructions everywhere, especially about what you’re not supposed to do. Can’t stray from the predetermined path. Can’t take photos in that direction. etc. etc.

We were surprised at the amount of meat, specifically pork, that is everywhere. Being a vegetarian or vegan must be pretty hard there. Another surprise - lack of bread (something hard to imagine for an Eastern European). Food is generally rice, pork, seafood and kimchi, and fairly spicy.

Photo Impressions

Almost all photos are taken by my wife. Equipment: Canon EOS 70D with Canon 24-70mm f/2.8 L II and Sigma 8-16mm f/4.5-5.6. Some taken by iPhone 6 or iPhone SE.

Seoul, Samcheong-Dong 삼청동

The area where we stayed in a hanok village, close the old imperial palaces. Narrow, steep streets. Mountains in the background on one side, highrise glass buildings on the other side. And churros that are better than a boyfriend!

Seoul, Bugaksan mountain 북악산

The hiking trail goes along the old city wall, right north to the Blue House (residence of the president). Probably for that reason you need to fill out registration form, there are guards every hundred meters, and a few layers of barbed wire, cameras and stuff. I got used to not having all that back home!

Seoul, Cheonggyecheon stream 청계천

Very nice walk in the center of Seoul. Great contrast of the nature & surrounding highrise.

Sokcho 속초

Went to Sokcho for a hike in Seoraksan national park nearby. After Seoul, this felt small and a bit empty? That said, after Seoul a lot of places can feel that way :) Apparently a lot of people go to Sokcho for the seafood, but none of us are big seafood fans, so that part was out. We walked around the seaside and went to the park the next day.

Seoraksan National Park 설악산국립공원

Sinheungsa temple (신흥사) is right at the entrance to the park.

We walked for couple hours in Ulsanbawi (울산바위) direction. Did not get to the actual rock formation since it’s a longer hike, and would probably be too hard for the kids.

Took a cable car towards Gwongeumseong fortress (설악산 권금성), around 800 meters elevation.

The photos above make it look as if there’s not that many people in the park. That was definitely not the case though. It was Saturday, and it was packed with visitors; apparently Koreans love walking or hiking in their national parks. We were very impressed by 2-4 year old kids walking and climbing stones on fairly long trails.

Gyeongju 경주

Used to be capital of the ancient Silla kingdom. A lot of tumuli and other olden things around.

There’s a big Bulguksa 불국사 temple nearby. Overall temples in Korea - at least the ones we’ve seen - feel more like museum sites than active temples. Very different impression from the buddhist temples that we saw in China, for example. South Korea is a fairly non-religious country, by the way.

Busan 부산

We liked Busan a lot! However the city is very spread out so getting from one place to another took a lot of time on the metro. We stayed in Haeundae area based on some internet advice.

The apartment we stayed in was impressively tiny! Which is fairly typical for the whole Korea, I guess.

Busan, Haedong Yonggungsa temple 해동 용궁사

An hour bus ride from Busan, a nice temple on the seaside.

Busan, Gamcheon Village 감천문화마을

OMG the streets are steep there! This was lovely, just take a local bus there instead of climbing all the way up from metro stop like we did :)

Jeju Island 제주도

Took a flight to Jeju from Busan Gimhae airport, and we stayed on the southern part.

Jeju is often listed as a “must visit” type of destination, but frankly we were not terribly impressed. Maybe because we made a mistake of thinking that we can get by public transportation like elsewhere in Korea – do not do that; just rent a car or hire a fulltime taxi! Oh right, I already mentioned this above… Overall, the whole place felt a bit of “too touristy” for our tastes at least.

The volcanic ash/rock formations - Yongmeori Coast (용머리해안) - near Sanbangsan Mountain (산방산) were pretty cool! There’s a Sanbanggulsa Temple (산방굴사) right there too.

Seoul, Itaewon 이태원

Flew back to Seoul Gimpo airport, and our last stay was in Itaewon district. Generally in Korea you see very few foreigners; not so in Itaewon! We had nice food, went to the N Tower and the Leeum museum. Was nice!

And then the next morning it was a long flight home!

I had some solar panels installed on my roof, so here’s a post with graphs & numbers & stuff!

TL;DR: should more or less cover my electricity usage; easy setup; more involved paperwork; cost about 5k€ (should get 30% of that back someday due to government incentives); expected ROI 10 years; panels themselves are less than half of the cost today.

Setup and Paperwork

Sometime this winter, I had a small array of 12 solar panels (about 20 square meters; 3.12kWp, “kWp” is maximum power rating in ideal conditions) setup on my roof. Besides the panels, a power inverter has to be set up (panels generate direct current). I had it slapped on a wall in my garage.

The setup was fairly impressive - basically one dude did it all by himself in six hours. Including getting the panels onto the roof. Efficient!

And then came the paperwork… That took about three months at Vogon pace; every little piece of paper had to be signed in triplicate, sent in, sent back, queried, lost, found, subjected to public inquiry, lost again, and finally buried in soft peat for three months and recycled as firelighters. Ok it’s not that bad; most of the paper busywork was done by the company that is doing the solar setup (I just had to sign an occasional document). But still, the whole chain of “setting up solar roof requires 20 paperwork steps” means that all these execute in a completely serial fashion and take three months.

Which means I could only “turn on” the whole thing a month ago.

How does it work?

The setup I have is a two-way electricity accounting system, where whatever the panels generate I can use immediately. If I need more power (or the sun is not shining), I use more from the regular grid. If I happen to generate more than I need, the surplus goes back into the grid for “storage” that I can recoup later (during the night, or during winter etc.). I pay about 0.03€ for kWh stored that way (regular grid electricity price is about 0.11€); the grid essentially works as a battery.

A two-way meter is setup by the energy company that tracks both the power used & power given back to the grid.

This setup is fairly new here in Lithuania; they only started doing this a couple years ago. I’ve no idea how it is in other countries; you’ll have to figure that out yourself!

Show me the graphs!

On a bright sunny day, the solar production graph almost follows the expected dot(N,L) curve :)

It generated a total of 21.4 kWh on that day. On a very cloudy/rainy day, the graph looks like this though (total 7.5 kWh generated):

My own usage averages to about 9 kWh/day, so output on a very cloudy summer day does not cover it :|

Over the course of May, daily production was on average 15.4 kWh/day:

For that month, from the 478 kWh produced I have returned/loaned/stored 371 kWh to the grid, and used 188 kWh from the grid during evenings.

All this is during near-midsummer sunny month, in a fairly northern (55°N) latitude. I’ll have to see how that works out during winter :)

What’s the cost breakdown?

Total cost of setting up everything was about 5200€:

• Solar panels 2120€ (40%)
• Power inverter 1310€ (25%)
• Design, paperwork, permits, all the Vogon stuff 800€ (15%)
• Construction work & transportation 680€ (14%)
• Roof fixture 300€ (6%)

So yeah, this means that even if future prices of the panels are expected to fall further, by itself that won’t affect the total cost that much. I would expect that future paperwork prices should fall down too, since today that process is way too involved & complexicated IMHO.

I should get about 30% of the cost back, due to some sort of government (or EU?) programs that cover that amount of solar (and I guess wind/hydro too) installations. But that will be more paperwork sometime later this year; will see.

What’s the ROI? Why do this?

Return on investment is something like 10 years, assuming roughly stable electricity prices, and if the solar modules retain 90% of their efficiency at the end of that.

Purely from financial perspective, it’s not an investment that would be worth the hassle, at this scale at least.

However, that assumes that the price I pay for electricity is the total actual cost of electricity. Which I think is not true; there’s a massive hidden cost that we all happily ignore – but is distributed to other poor people (or future ourselves) in terms of droughts, fires, floods, political instabilities, wars and so on. Solar has some hidden cost too (mining rare minerals needed for the solar modules; that is a finite resource and has long-lasting non-trivial effects on the places where mining happens), but it feels like today that’s a lesser evil than the alternative of coal/oil/gas.

So if I can do even a tiny part towards “slightly better balance”, I’m happy to do it. That’s it!

Recently someone at work said “hey Aras, you write great feature overview docs, what are the tips & tricks to do them” and that got me thinking… The only obvious one I have is:

Imagine what a user would want to know about {your thing}, and write it down.

I had some more detail in there, e.g. when writing a feature overview or “proposal for a thing”, write down:

1. How do things work right now, what are current workflows of achieving this and what are the problems with it.
2. How will this new feature solve those problems or help in other ways.
3. Write down what a user would need to know. Including use-cases, examples and limitations.

Now, that is simple enough. But looking at a bunch of initial docs, release notes, error messages, UI labels and other user-facing things, both at Unity and elsewhere, it’s presumably not that obvious to everyone :)

Random Examples

Release Notes

Mercurial DVCS used to have release notes that were incomprehensible (to me as a user at least). For example here, take a look at hg 3.4 notes. What does “files: use ctx object to access dirstate” or “dirstate: fix order of initializing nf vs f” mean? Are these even words?!

Even git, which I think is a somewhat Actively Hostile To The User^(*) version control system, had better release notes. Check out git 2.12.0 notes for example.

(*) I know, “Yeah, well, that’s just, like, your opinion, man” etc.

Thankfully, since Mercurial 3.7 they started writing “feature overview” pages for each release, which helps a lot to point out major items in actual human language. Now I can read it and go “oh this looks sweet, I should upgrade” instead of “I know the words, but the sentences don’t mean anything. Eh, whatever!”.

Yes, this does mean that you can’t just dump a sorted list of all source control commit messages and call them release notes. Someone has to sit down and write an overview. However, your users will thank you for that!

Pull Requests

A pull request or a code review request asks other people to look at your code. They have to spend their time doing that! Time they could perhaps spend doing something else. If you can spend five minutes making their job easier, that is often very much worth it.

I used to review a lot of code in 2015/2016 (~20 PRs each week), and the “ugghhh, not this again” reaction was whenever I saw pull requests like this. In each case, that single sentence is the only thing in PR description, with no further info besides list of commits & code diff:

• “Updates to 2D framework + all new animation window full of win”. 374 commits, 100 files changed, 8000 lines of code.
• “Latest MT rendering work”. 64 commits, 95 files, 3000 lines of code.
• “Multithreaded rendering refactor”. 119 commits, 79 files, 4000 lines of code.
• “UWP Support”. 224 commits, 219 files, 7000 lines of code.
• “Graphics jobs (preliminary)”. 114 commits, 112 files, 2000 lines of code.

Seriously. You just spent several months doing all that work, and have one sentence to describe it?!

Sometimes just a list of commit messages is enough to describe pull request contents for the reviewers (this is mostly true for PRs that are a bunch of independent fixes; with one commit per fix). If that is the case, say that in the PR description! The above list of PR examples were very much not this case though :)

What would be good PR descriptions that make reviewer’s job significantly easier? Here’s some good ones I’ve seen:

How do you make more PRs have good descriptions?

Often I would go and poke PR authors asking for the description, especially for large ones. Something like “Wow this is a big PR! Can you write up a summary of changes in the description; would make reviewing it much easier”.

Another thing we did was make our pull request tool pre-fill the PR description field:

Purpose of this PR:

[Desc of feature/change. Links to screenshots, design docs, user docs, etc. Remember reviewers may be outside your team, and not know your feature/area that should be explained more.]

Testing status:

[Explanation of what’s tested, how tested and existing or new automation tests. Can include manual testing by self and/or QA. Specify test plans. Rarely acceptable to have no testing.]

Technical risk:

[Overall product level assessment of risk of change. Need technical risk & halo effect.]

Comments to reviewers:

[Info per person for what to focus on, or historical info to understand who have previously reviewed and coverage. Help them get context.]

This simple change had a massive impact on quality of PR descriptions. Reviewers are now more happy and less grumpy!

Commit Messages and Code Comments

A while ago I was looking at some screen resolution handling code, and noticed that some sequence of operations was done in a different order on DX11 compared to DX9 or OpenGL. Started to wonder why, turns out I myself have changed it to be different… five years ago… with a commit message that says “fixed resolution switches” (100 lines of code changed).

Bad Aras, no cookie!

Five years later (of heck, even a few months later) you yourself will not remember what case exactly this was fixing. Either add that info into the commit message, or write down comments on tricky parts of code. Or both. Especially in code that is known to be hard to get right (and anything involving resolution switches on Windows is). Yes, sometimes it does make sense to write ten lines of comments for a single line of code. Future you, or your peers, or anyone who will look into this code will thank you for that.

I know that since I have had “pleasure” of maintaining my own code ten years later :) Cue “…and if you tell that to the young people today, they won’t believe you…” quote.

Any time someone might be wondering “why is this done?” about a piece of code, write it down. My own code these days often has comments like:

// All "folders" in the solution are represented as "fake projects" too;
// with a special parent GUID and instead of pointing to vcxproj files they
// just point to their own names.
var foldersParentGuid = "2150E333-8FDC-42A3-9474-1A3956D46DE8";

or

// VS filters file needs filter elements to exist for the whole folder hierarchy
// (e.g. if we have "Editor\Platform\Windows", we need to have "Editor\Platform",
// and "Editor" too). Otherwise it will silently skip grouping files into filters
// and display them at the root.
var parentFolders = allFolders.SelectMany(f => f.RecursiveParents).Where(p => p.Depth > 0).ToArray();
allFolders.UnionWith(parentFolders);

// VS seems to load projects a bit faster if file/folder lists are sorted
var sortedFolders = allFolders.OrderBy(f => f); 

If anything else, I write these for future myself.

Empathy

What all the above has in common?

Empathy.

Putting yourself into position of the reader, user, colleague, future self, etc. What do they want to know? What makes their job/task/life easier?

Mikko Mononen once showed these amazing resources:

• The Paradox of Empathy by Scott Jenson.
• Gel 2009 talk by magician Jamy Ian Swiss.

I’m not sure what conclusion I wanted to make, so, ehhh, that’s it. Read the resources above!

Everyone using Visual Studio is probably familiar with these dropdowns, that contain build configurations (Debug/Release is typical) and platforms (Win32/x64 is typical):

When opening a fresh new solution (.sln) file, which ones does Visual Studio pick as default? The answer is more complex than the question is :)

After you have opened a solution and picked a config, it is stored in a hidden binary file (VS2010: {solutionname}.suo, VS2015: .vs/{solutionname}/v14/.suo) that contains various user-machine-specific settings, and should not be put into version control. However what I am interested is what is the default configuration/platform when you open a solution for the first time.

Default Platform

Platform names in VS solution can be arbitrary identifiers, however platform names defined in the project files have to match an installed compiler toolchain (e.g. Win32 and x64 are toolchain names for 32 and 64 bit Windows on Intel CPUs, respectively).

Turns out, default platform is first one from an alphabetically, case insensitive, sorted list of all solution platforms.

This means if you have Win32 and x64 as the solution platforms, then 32 bit one will be the default. That probably explains why in recent VS versions (at least since 2015), the built-in project creation wizard started naming them as x86 and x64 instead – this conveniently makes x64 be default since it sorts first.

A note again, platform names in the project have to be from a predefined set; so they have to stay Win32 and x64 and so on. Even if you use VS for editing files in a makefile-type project that invokes compiler toolchains that VS does not even know about (e.g. WebGL – for that project, you still have to pick whether you want to name the platform as Win32 or x64).

Default Configuration

When you have Debug and Release configurations, VS picks Debug as the default one. What if you have more configurations, that are more complex names (e.g. we might want to have a Debug WithProfiler LumpedBuild)? Which one will be the default one?

So, a pop quiz time! If all projects in the solution end up having this set of configurations, which one will VS use by default?

Foo
Foo_Bar
Foo Bar
Alot
AlotOfBanjos
Alot Of Bees
Debug
Debug_Lumped
Debug_Baloney
DebugBaloney

You might have several guesses, and they all would make some sense:

• Foo since it’s the first one in the solution file,
• Alot since it’s the first one alphabetically (and hey that’s how VS chooses default platform),
• Debug since VS probably has some built-in logic to pick “debug” first.

Of course all these guesses are wrong! Out of the list above, VS will pick Debug_Baloney as the default one. But why?!

The logic seems to be something like this (found in this stackoverflow answer, except it needed an addition for the underscore case). Out of all configurations present:

1. Sort them (almost) alphabetically, case insensitive,
2. But put configs that start with debug before all others,
3. Also config that is another one with a Whatever or _Whatever added, go before it. So A_B goes before A; Debug All goes before Debug (but DebugAll goes after Debug).
4. And now pick the first one from the list!

I do hope there is a good explanation for this, and today VS team probably can’t change it because doing so would upset three bazillion existing projects that learned to accidentally or on purpose to depend on this.

However this means that today in our code I have to write things like this:

// We want to put spaces into default config of the 1st native program;
// see ConfigurationToVSName. Default configs in each programs might be
// separate sets, so e.g. in the editor there might be "debug_lump" and
// the standalone might only have a longer one "debug_lump_il2cpp", and
// if both were space-ified VS would pick "debug lump il2cpp" as default.
var defaultSolutionConfig = nativePrograms[0].ValidConfigurations.Default;
GenerateSolutionFile(nativePrograms, solutionGuid, projectGuids, defaultSolutionConfig);
GenerateHelperScripts();
foreach (var np in nativePrograms)
{
    GenerateProjectFile(np, projectGuids, defaultSolutionConfig);
    GenerateFiltersFile(np);
}

and this:

// Visual Studio has no way to indicate which configuration should be default
// in a freshly opened solution, but uses logic along the lines of
// http://stackoverflow.com/a/41445987 to pick default one:
//
// Out of all configurations present:
// 1. Sort them (almost) alphabetically, case insensitive,
// 2. But put configs that start with "debug" before all others,
// 3. Also config that is another one with a " Whatever" or "_Whatever" added,
//    go before it. So "A_B" goes before "A"; "Debug All" goes before "Debug"
//    (but "DebugAll" goes after "Debug").
// 4. And now pick the first one from the list!
//
// Our build configs are generally underscore-separated things, e.g. "debug_lump_il2cpp".
// To make the default config be the first one, replace underscores in it with
// spaces; that will make it sort before other things (since space is before
// underscore in ascii), as long as it starts with "debug" name.
string ConfigurationToVSName(CApplicationConfig config, CApplicationConfig defaultConfig)
{
    if (config.IdentifierNoPlatform != defaultConfig.IdentifierNoPlatform)
        return config.IdentifierNoPlatform;
    return config.IdentifierNoPlatform.Replace('_', ' ');
}

This is all trivial code of course, but figuring out the logic for what VS ends up doing did took some experimentation. Oh well, now I know! And you know too, even if you never wanted to know :)

And voilà, debug_lump is picked by VS as the default in our auto-generated project files, which is what I wanted here. Without any extra logic, it was picking debug_lump_dev_il2cpp since that sorts before debug_lump as per rules above.

That’s it for now!