Saturday, August 22, 2015

Objective measure of resources

Intro


A recent post by a friend got me thinking: https://plus.google.com/+DarkGriffin/posts/Uv115o7hPyG

Part I - Theory

There should be an *objective* measure of whether a resource like this any good...when compared to other resources of similar technicality. That caveat is so that you can't compare a collection of 3D art to a collection of png sprites, or something - ideally for the comparison the resources must be directly comparable.

Theoretically, even minor technical dissimilarities (such as png vs svg), or even picture size, such as 1024x800 vs 1440x1200 should count against comparing two resources, except where it can be shown that the resources are technically similar (svg vs ai format, or you include both png and svg, or you have many types/sizes of images, or maybe your technical resources contain both textures (images) and 3d models - although it should note that the technical similarity should be as close as possible).

Moving on, if a resource is technically similar to another one, then I propose a basic objective metric should apply - quantity vs quality.

i.e., a good resource should have a high, high number of 'units' and categories which are logically independent, and should be populous, i.e. there should be many similar but slightly dissimilar pieces.

If you have a lower number of units, then your resource is necessarily sparse. If you have a high number of resources, but they are all highly logically correlated, then you have extremely low diversity (maybe your resource is a bunch of cat pictures). Finally, even if you have a high number of resources, you do not want a 'single re-presenter per bucket', i.e. the point at which you are flooded with categorical examples (so you have one of each type of animal for example), but few representations of each animal.

Since, naturally, 'larger' libraries of resources should have 'accidentally' larger numbers of categories and representations, it should be possible to compare a large resource to a small resource for the metric of quality, i.e. what is the ratio of categories to representations, and for resources with with other varying characteristics, where one part is greater than the other, such as one collection with a greater collection of categories than the other other, but a difference in representation and a similarity in size, this ratio should make it possible to objectively compare otherwise disparate and seemingly incomparable resources.

Finally, I propose that there is one final metric with which to compare two resources - market relevance. In order to apply this metric, as with the technical reference, it should only be applied to markets which are extremely similar (photography databases and iconography make poor comparisons, for example).

Specifically, it is like so - if a resource contains a relatively 'high' number of 'popular' resources compared to another resource, it should be worth more than a resource with a good category/representation ratio, but a low 'popularity' relevance - it is possible perhaps that the most useful to a video game market in iconography is e.g. weaponry, and while one resource provides an abundance of flowers, cooking apparel, and small creatures, it has a low number of weaponry, while the second resource which contains only say weaponry and cars and a small number of creatures, flowers, and cooking apparel, has much more inherent worth.

So I propose, to say objectively whether a resource is good or not, you should compare it with a set of other resources, and provide the following comparison:

representation/categories ratio (higher is better)
category count comparison
size comparison
utility/size ratio (higher is better, utility is out of 1, i.e. the best possible scenario is that every image is used)

Perhaps this is another comparison:
'untapped potential' comparison - utility/size ratio is similar, however one resource contains considerably more resources than another, i.e. there are more unharnessed patterns and therefore more inherent worth per item.

Part II - Back of the envelope application

And now for a bit of hand waving because I am a bit too lazy to actually crunch the numbers properly...this collection contains 102 self-styled 'categories', with it looks like on average 30 representations per category...by that alone, it looks, decent, however we are lacking anything to compare it with:

https://www.iconfinder.com/free_icons/

Maybe not the best resource in the world, however it was one of the first off the search engine. So it can't be that bad. Nevertheless, I'm not going to use it, because it seems heavily focused on web-icons, whereas yours was more 'game icons'. So I would say it is technically dissimilar.

http://www.flaticon.com/packs/

After looking a bit harder, I found this.

939 'icon packs', i.e. categories, with on average 60 (I'm ballparking) icons per category...so 0.06 ratio

compared to 30/102 - 0.29

102 vs 939 - its clear who the winner is here...
size comparison - 60*939 ~= 10k, roughly 10x the size of this collection...
utility/size ratio - this is the interesting comparison, looking briefly at the game-icons.net site, there were a high number of 'gamey' icons such as e.g. weapons (some categories here of 100 vs the average of 30, probably 2std devs away from the average and hence enough to be significant).

The problem being, there were maybe 1-2 of these 'utility' categories, and the rest were all what looked like...junk. Worse than that, some of these categories were broad, and when you looked at them, the icons within were highly dissimilar to the point that you could question if they really even belonged to the same category.

On the other hand, flaticon had  many categories which definitely looked useful to game design. There is definitely a so-called Big Data problem hidden in here, as to determining which icons were better correlated with their categories and how relevant they are to game design.

However, given that there so much doubt, from a quick glance, between measuring the metrics of flaticon vs game-icon.net, I'm skeptical that game-icon's quality is much better than that of flaticon's.

No comments:

Post a Comment

print "#3110 vv021|)"