Warning: patented denise tl;dr "it's a lot more complicated than even the complicated version" meandering ahead, heh.
It's the same problem that we're all struggling with in content moderation/Trust and Safety as a profession: starting at levels of "content submitted per hour" that are, like, barely 2x or 3x as much as we get, the process of both detecting and preventing automated abuse such as spam and the process of handling violations of the Terms of Service/getting really bad shit off your site starts to be both a significant undertaking and extremely traumatic to any humans involved. (I've written extensively about the psychological harm that content moderation work has on the people who do it, because I was one of the people doing it in that first wave of user-generated content sites before we had any idea what the psychological consequences of it were, and I have lost so many of my contemporaries and professional acquaintances from the early days to death by suicide or misadventure. And that's not counting how many literally renounced technology and online communications and are living off grid and homesteading or small farming! Which is a very common exit profession for tech.)
So, good automation detection/classification tools, powered by machine learning (and these days, large language model based ML is the most effective form of machine learning for this) not only gets dangerous content taken down faster, becauss you don't have to wait for a human to report it and another human to take action, it also saves a lot of human suffering and trauma. I will refrain from going into a ton of detail except to say that automated image content analysis and detection is probably the single thing that has prevented more moderator trauma than any single other mitigation in the history of Trust and Safety... except. General purpose LLMs are bad at this and you need custom trained models, and a lot of the custom trained models are developed through the outsourcing of labeling/classification labor to the developing world where labor costs are lower (and often to where employee occupational-injury protection laws are weaker or nonexistent), and a shitload of it is subcontracted and abstracted away even further to companies that are highly incentivized to cut their costs and increase their throughput as aggressively as possible and the worker psychological protections we do know work a little cost more than just burning through traumatized people, so the data workers who are classifying the training data don't even have the basic psychological mitigations we know prevent some of the trauma and moral injury from constant immersion in the worst humanity has to offer. And now we're, like, at least four layers of subcontracting away from what "someone who just wants to use automated detection so their moderators don't have to look at child sex abuse and snuff films all day" has the power to influence, control, or mandate better working conditions.
So to some extent it is outsourcing trauma work (and Trust and Safety is inherently trauma work) to people in developing countries! I have a gigantic fucking problem with that! But someone has to do that trauma work, because "don't moderate your website" is not an option, and training a machine learning model to do it will protect more people overall just because of the sheer volume of content posted to the internet every day, and a lot of places are already outsourcing their human moderation trauma work to layers of providers that end up with the trauma work being done by people in developing countries with shit worker protections anyway, so is it better to traumatize people on an ongoing basis instead of use the tools that were developed by traumatizing people? Because we've tried really hard as an industry to come up with a solution that involves no traumatizing people at all, but we haven't found it yet and given human nature we're never gonna! As much as I hate pure utilitarian ethical arguments because they're so easily twisted into justifying atrocities, and prefer my ethical analysis to include a balance of deontological and consequential ethical principles, that one is genuinely difficult to reason through no matter what ethical framework you apply!
And that's just one small element of many. Basically, there's a reason I say my undergraduate degree in philosophy and history of religion is significantly more applicable and valuable to my work in social media product development and Trust and Safety than I ever possibly could have expected, even applying the "the liberal arts are always more applicable than you think", because dear God all of this shit is incredibly complicated and even more nuanced than anyone would think.
But we set out our guiding ethical principles for Dreamwidth when we started and we come back to them constantly to guide our thinking on this, and they haven't changed. Our purpose is to provide our users the tools they need and want to communicate however and whatever they want to without having to sacrifice their privacy and security in giving away their data. We exist to facilitate human conversation, connection, and discussion without the distorting influence of surveillance capitalism (and partially to serve as proof it's possible and inspire more people to try it -- there's a reason I offer free consultation to anyone who's starting or thinking of starting an independent social media site, because I want there to be a million more of them!) Anything we do, any tool we chose, any feature we implement, any process we put in place, is in service to those goals, and we examine everything we do inside that framework.
Sometimes we do use tools and/or third party providers that are less awesome than we would like! Example: we really do need a lot of the "audience device technical capacity" features of Google Analytics in order to determine how to best prioritize development and bug fixing time, and nothing self-hosted has all the functionality we need (and self-hosting a tool increases complexity, in that now you have to keep up that tool's dependencies, watch for security flaws, etc: supply chain attacks where someone compromises a dependency are accelerating.) On the other hand: it's Google. On the other hand, that aggregate information surfaces important things like "what assistive technology people are using to access the site, so we know what to test in". And so on, ad infinitum.
So we do the analysis every time, dig through all the possible risks and threats, figure out whether we can mitigate them or not -- I'm really good at redteaming how something could be exploited by now! For Google Analytics, for instance, we only include it on a representative subset of site pages and never put it in user journal space (unless the user wants to use their GA account because they're comfortable with it). For the implementation of our payment system, which we have to outsource, we structured our implementation and integration so that it would be impossible for a subpoena to our payment processor (who's way less likely to fight a bullshit subpoena than we would be, because we are very fighty) to associate someone's payment with their DW username, and impossible for any theoretical security breach on DW to give an intruder the ability to look at a payment on DW and get someone's government identity from it. (There's a limit to how far that can go with protecting payment identity from being associated with a username from a government subpoena to us, because a court does have the ability to command us to go get the data from the payment processor ourselves and provide it, but see above re: us being fighty about that. It's never happened, to be clear, but we would fight the shit out of any bullshit identity/payment data subpoena we got just like we'd fight the shit out of any other subpoena that we didn't think was necessary, appropriate, proportionate, and addressing an extremely serious crime such as child sexual abuse or genuine human trafficking.)(And just to be absolutely crystal clear: I mean actual coercive violence-backed human labor, sexual, medical, etc trafficking, not "all consensual and uncoerced sex work is actually human trafficking even if that implication means you're the person who's trafficking yourself", because too many people are trying to use the language of human trafficking to make a lot of perfectly fine things sound scary.)
The tl;dr (way, way too late): the usual exhaustive comments explaining a million bits of nuance you're used to seeing from me are the short version of the analysis we do on our decisions! I'm usually skipping over a whole lot of shit! But we always come back to those guiding principles, and we always aim for, does this thing improve our ability to facilitate human communication without affecting user privacy or security, and what guardrails do we need to put on it so that if someone not-us in the chain fucks up, gets compromised, or turns malicious, the exposure to our users will be prevented as much as possible?
no subject
Warning: patented![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
denise tl;dr "it's a lot more complicated than even the complicated version" meandering ahead, heh.
It's the same problem that we're all struggling with in content moderation/Trust and Safety as a profession: starting at levels of "content submitted per hour" that are, like, barely 2x or 3x as much as we get, the process of both detecting and preventing automated abuse such as spam and the process of handling violations of the Terms of Service/getting really bad shit off your site starts to be both a significant undertaking and extremely traumatic to any humans involved. (I've written extensively about the psychological harm that content moderation work has on the people who do it, because I was one of the people doing it in that first wave of user-generated content sites before we had any idea what the psychological consequences of it were, and I have lost so many of my contemporaries and professional acquaintances from the early days to death by suicide or misadventure. And that's not counting how many literally renounced technology and online communications and are living off grid and homesteading or small farming! Which is a very common exit profession for tech.)
So, good automation detection/classification tools, powered by machine learning (and these days, large language model based ML is the most effective form of machine learning for this) not only gets dangerous content taken down faster, becauss you don't have to wait for a human to report it and another human to take action, it also saves a lot of human suffering and trauma. I will refrain from going into a ton of detail except to say that automated image content analysis and detection is probably the single thing that has prevented more moderator trauma than any single other mitigation in the history of Trust and Safety... except. General purpose LLMs are bad at this and you need custom trained models, and a lot of the custom trained models are developed through the outsourcing of labeling/classification labor to the developing world where labor costs are lower (and often to where employee occupational-injury protection laws are weaker or nonexistent), and a shitload of it is subcontracted and abstracted away even further to companies that are highly incentivized to cut their costs and increase their throughput as aggressively as possible and the worker psychological protections we do know work a little cost more than just burning through traumatized people, so the data workers who are classifying the training data don't even have the basic psychological mitigations we know prevent some of the trauma and moral injury from constant immersion in the worst humanity has to offer. And now we're, like, at least four layers of subcontracting away from what "someone who just wants to use automated detection so their moderators don't have to look at child sex abuse and snuff films all day" has the power to influence, control, or mandate better working conditions.
So to some extent it is outsourcing trauma work (and Trust and Safety is inherently trauma work) to people in developing countries! I have a gigantic fucking problem with that! But someone has to do that trauma work, because "don't moderate your website" is not an option, and training a machine learning model to do it will protect more people overall just because of the sheer volume of content posted to the internet every day, and a lot of places are already outsourcing their human moderation trauma work to layers of providers that end up with the trauma work being done by people in developing countries with shit worker protections anyway, so is it better to traumatize people on an ongoing basis instead of use the tools that were developed by traumatizing people? Because we've tried really hard as an industry to come up with a solution that involves no traumatizing people at all, but we haven't found it yet and given human nature we're never gonna! As much as I hate pure utilitarian ethical arguments because they're so easily twisted into justifying atrocities, and prefer my ethical analysis to include a balance of deontological and consequential ethical principles, that one is genuinely difficult to reason through no matter what ethical framework you apply!
And that's just one small element of many. Basically, there's a reason I say my undergraduate degree in philosophy and history of religion is significantly more applicable and valuable to my work in social media product development and Trust and Safety than I ever possibly could have expected, even applying the "the liberal arts are always more applicable than you think", because dear God all of this shit is incredibly complicated and even more nuanced than anyone would think.
But we set out our guiding ethical principles for Dreamwidth when we started and we come back to them constantly to guide our thinking on this, and they haven't changed. Our purpose is to provide our users the tools they need and want to communicate however and whatever they want to without having to sacrifice their privacy and security in giving away their data. We exist to facilitate human conversation, connection, and discussion without the distorting influence of surveillance capitalism (and partially to serve as proof it's possible and inspire more people to try it -- there's a reason I offer free consultation to anyone who's starting or thinking of starting an independent social media site, because I want there to be a million more of them!) Anything we do, any tool we chose, any feature we implement, any process we put in place, is in service to those goals, and we examine everything we do inside that framework.
Sometimes we do use tools and/or third party providers that are less awesome than we would like! Example: we really do need a lot of the "audience device technical capacity" features of Google Analytics in order to determine how to best prioritize development and bug fixing time, and nothing self-hosted has all the functionality we need (and self-hosting a tool increases complexity, in that now you have to keep up that tool's dependencies, watch for security flaws, etc: supply chain attacks where someone compromises a dependency are accelerating.) On the other hand: it's Google. On the other hand, that aggregate information surfaces important things like "what assistive technology people are using to access the site, so we know what to test in". And so on, ad infinitum.
So we do the analysis every time, dig through all the possible risks and threats, figure out whether we can mitigate them or not -- I'm really good at redteaming how something could be exploited by now! For Google Analytics, for instance, we only include it on a representative subset of site pages and never put it in user journal space (unless the user wants to use their GA account because they're comfortable with it). For the implementation of our payment system, which we have to outsource, we structured our implementation and integration so that it would be impossible for a subpoena to our payment processor (who's way less likely to fight a bullshit subpoena than we would be, because we are very fighty) to associate someone's payment with their DW username, and impossible for any theoretical security breach on DW to give an intruder the ability to look at a payment on DW and get someone's government identity from it. (There's a limit to how far that can go with protecting payment identity from being associated with a username from a government subpoena to us, because a court does have the ability to command us to go get the data from the payment processor ourselves and provide it, but see above re: us being fighty about that. It's never happened, to be clear, but we would fight the shit out of any bullshit identity/payment data subpoena we got just like we'd fight the shit out of any other subpoena that we didn't think was necessary, appropriate, proportionate, and addressing an extremely serious crime such as child sexual abuse or genuine human trafficking.)(And just to be absolutely crystal clear: I mean actual coercive violence-backed human labor, sexual, medical, etc trafficking, not "all consensual and uncoerced sex work is actually human trafficking even if that implication means you're the person who's trafficking yourself", because too many people are trying to use the language of human trafficking to make a lot of perfectly fine things sound scary.)
The tl;dr (way, way too late): the usual exhaustive comments explaining a million bits of nuance you're used to seeing from me are the short version of the analysis we do on our decisions! I'm usually skipping over a whole lot of shit! But we always come back to those guiding principles, and we always aim for, does this thing improve our ability to facilitate human communication without affecting user privacy or security, and what guardrails do we need to put on it so that if someone not-us in the chain fucks up, gets compromised, or turns malicious, the exposure to our users will be prevented as much as possible?