InfoWorld reports: Don't look now, but 25% of organizations surveyed in the United Kingdom have already moved half or more of their cloud-based workloads back to on-premises infrastructures. This is according to a recent study by Citrix, a Cloud Software Group business unit. The survey questioned 350 IT leaders on their current approaches to cloud computing. The survey also showed that 93% of respondents had been involved with a cloud repatriation project in the past three years. That is a lot of repatriation. Why?

Security issues and high project expectations were reported as the top motivators (33%) for relocating some cloud-based workloads back to on-premises infrastructures such as enterprise data centers, colocation providers, and managed service providers (MSPs). Another significant driver was the failure to meet internal expectations, at 24%... Those surveyed also cited unexpected costs, performance issues, compatibility problems, and service downtime. The most common motivator for repatriation I've been seeing is cost. In the survey, more than 43% of IT leaders found that moving applications and data from on-premises to the cloud was more expensive than expected.

Although not a part of the survey, the cost of operating applications and storing data on the cloud has also been significantly more expensive than most enterprises expected. The cost-benefit analysis of cloud versus on-premises infrastructure varies greatly depending on the organization... The cloud is a good fit for modern applications that leverage a group of services, such as serverless, containers, or clustering. However, that doesn't describe most enterprise applications.
The article cautions, "Don't feel sorry for the public cloud providers."

"Any losses from repatriation will be quickly replaced by the vast amounts of infrastructure needed to build and run AI-based systems... As I've said a few times here, cloud conferences have become genAI conferences, which will continue for several years."

An anonymous reader shared this report from Reuters: Nvidia is building a new business unit focused on designing bespoke chips for cloud computing firms and others, including advanced AI processors, nine sources familiar with its plans told Reuters. The dominant global designer and supplier of AI chips aims to capture a portion of an exploding market for custom AI chips and shield itself from the growing number of companies pursuing alternatives to its products.

The Santa Clara, California-based company controls about 80% of high-end AI chip market, a position that has sent its stock market value up 40% so far this year to $1.73 trillion after it more than tripled in 2023. Nvidia's customers, which include ChatGPT creator OpenAI, Microsoft, Alphabet, and Meta Platforms, have raced to snap up the dwindling supply of its chips to compete in the fast-emerging generative AI sector. Its H100 and A100 chips serve as a generalized, all-purpose AI processor for many of those major customers. But the tech companies have started to develop their own internal chips for specific needs. Doing so helps reduce energy consumption, and potentially can shrink the cost and time to design.

Nvidia is now attempting to play a role in helping these companies develop custom AI chips that have flowed to rival firms such as Broadcom and Marvell Technology, said the sources, who declined to be identified because they were not authorized to speak publicly...

Nvidia moving into this territory has the potential to eat into Broadcom and Marvell sales.

Longtime Slashdot reader Alain Williams shares a report from the BBC: Users of Ring video doorbells have reacted angrily to a huge price hike being introduced in March. After buying the devices, customers can pay a subscription to store footage on the cloud, download clips and get discounted products. That subscription is going up 43%, from $44 to $63 per device, per year, for basic plan customers. The firm, which is owned by Amazon, insisted it still provided "some of the best value in the industry." Its customers appear not to to agree.

Google has renamed its AI assistant to "Gemini" and unveiled a paid subscription tier offering. The $19.99/month "Gemini Advanced" includes a more powerful AI model and cloud storage integration, targeting users seeking advanced content creation and complex query resolution. Google is also leveraging its Android user base by making Gemini the default digital assistant, aiming to replicate the success of its billion-user products.

"When Satya Nadella replaced Steve Ballmer as Microsoft CEO in February 2014, the software company was mired in mediocrity," writes CNBC, noting that Microsoft's market cap was just over $300 billion.

"A decade later, Microsoft's valuation has swelled tenfold, to $3.06 trillion, making it the world's most valuable public company, ahead of Apple." (And it's also "firmly entrenched as a leader in key areas, such as cloud and artificial intelligence.") As Nadella marks his 10-year anniversary at the helm, he's widely praised across the tech industry for changing the narrative at Microsoft, whose stock fell 30% during Ballmer's 14 years at the top. In that era, the company was squelched by Google in web search and mobile and was completely left behind in social media. Many tech industry analysts and investors would say that, thanks largely to Nadella, Microsoft is now set up to be a powerhouse for the foreseeable future...

In a 2020 interview, Pat Gelsinger, then CEO of VMware, said offering his company's software on Microsoft's Azure cloud was akin to a "Middle East peace treaty...." In the Nadella age, Microsoft has also contributed to open-source projects, released software under open-source licenses and released a version of its Teams communications app for Linux... In 2018, Nadella came to believe in the idea of buying GitHub just 20 minutes after Nat Friedman, then a Microsoft corporate vice president, started pitching him on it. Right away, Nadella suggested that Friedman become GitHub's new CEO, Friedman said. Microsoft paid $7.5 billion for the code-storage startup...

While Nadella may not bring as much entertainment value, he's proven to be more effective than Ballmer when it comes to dealmaking. In addition to GitHub, Nadella has made pricey acquisitions such as LinkedIn, Minecraft parent Mojang, and Nuance Communications that have contributed to Microsoft's top line. More recently, Nadella helped Microsoft land the $75 billion acquisition of game publisher Activision Blizzard...
The article also adds that Microsoft "looked at buying TikTok in the U.S. in 2020, but nothing came of those discussions."

Slashdot reader Unpopular Opinions requests suggestions from the Slashdot community: Lately a boom of companies decided to play their "nice guy" card, providing us with a trove of information about our own sites, DNS servers, email servers, pretty much anything about any online service you host.

Which is not anything new... Companies have been doing this for decades, except as paid services you requested. Now the trend is basically anyone can do it over my systems, and they are always more than happy to sell anyone, me included, my data they collected without authorization or consent. It's data they never had the rights to collect and/or compile to begin with, including data collected thru access attempts via known default accounts (Administrator, root, admin, guest) and/or leaked credentials provided by hacked databases when a few elements seemingly match...

"Just block those crawlers"? That's what some of those companies advise, but not only does the site operator have to automate it themself, not all companies offer lists of their source IP addresses or identify them. Some use multiple/different crawler domain names from their commercial product, or use cloud providers such as Google Cloud, AWS and Azure â" so one can't just block access to their company's networks without massive implications. They also change their own information with no warning, and many times, no updates to their own lists. Then, there is the indirect cost: computing cost, network cost, development cost, review cycle cost. It is a cat-and-mice game that has become very boring.

With the raise of concerns and ethical questions about AI harvesting and learning from copyrighted work, how are those security companies any different from AI, and how could one legally put a stop on this?
Block those crawlers? Change your Terms of Service? What's the best fix... Share your own thoughts and suggestions in the comments.

How can you stop security firms from harvesting your data?

An anonymous reader quotes a report from ProPublica: Over the last decade, police departments across the U.S. have spent millions of dollars equipping their officers with body-worn cameras that record what happens as they go about their work. Everything from traffic stops to welfare checks to responses to active shooters is now documented on video. The cameras were pitched by national and local law enforcement authorities as a tool for building public trust between police and their communities in the wake of police killings of civilians like Michael Brown, an 18 year old black teenager killed in Ferguson, Missouri in 2014. Video has the potential not only to get to the truth when someone is injured or killed by police, but also to allow systematic reviews of officer behavior to prevent deaths by flagging troublesome officers for supervisors or helping identify real-world examples of effective and destructive behaviors to use for training. But a series of ProPublica stories has shown that a decade on, those promises of transparency and accountability have not been realized.

One challenge: The sheer amount of video captured using body-worn cameras means few agencies have the resources to fully examine it. Most of what is recorded is simply stored away, never seen by anyone. Axon, the nation's largest provider of police cameras and of cloud storage for the video they capture, has a database of footage that has grown from around 6 terabytes in 2016 to more than 100 petabytes today. That's enough to hold more than 5,000 years of high definition video, or 25 million copies of last year's blockbuster movie "Barbie." "In any community, body-worn camera footage is the largest source of data on police-community interactions. Almost nothing is done with it," said Jonathan Wender, a former police officer who heads Polis Solutions, one of a growing group of companies and researchers offering analytic tools powered by artificial intelligence to help tackle that data problem.

The Paterson, New Jersey, police department has made such an analytic tool a major part of its plan to overhaul its force. In March 2023, the state's attorney general took over the department after police shot and killed Najee Seabrooks, a community activist experiencing a mental health crisis who had called 911 for help. The killing sparked protests and calls for a federal investigation of the department. The attorney general appointed Isa Abbassi, formerly the New York Police Department's chief of strategic initiatives, to develop a plan for how to win back public trust. "Changes in Paterson are led through the use of technology," Abbassi said at a press conference announcing his reform plan in September, "Perhaps one of the most exciting technology announcements today is a real game changer when it comes to police accountability and professionalism." The department, Abassi said, had contracted with Truleo, a Chicago-based software company that examines audio from bodycam videos to identify problematic officers and patterns of behavior.

For around $50,000 a year, Truleo's software allows supervisors to select from a set of specific behaviors to flag, such as when officers interrupt civilians, use profanity, use force or mute their cameras. The flags are based on data Truleo has collected on which officer behaviors result in violent escalation. Among the conclusions from Truleo's research: Officers need to explain what they are doing. "There are certain officers who don't introduce themselves, they interrupt people, and they don't give explanations. They just do a lot of command, command, command, command, command," said Anthony Tassone, Truleo's co-founder. "That officer's headed down the wrong path." For Paterson police, Truleo allows the department to "review 100% of body worn camera footage to identify risky behaviors and increase professionalism," according to its strategic overhaul plan. The software, the department said in its plan, will detect events like uses of force, pursuits, frisks and non-compliance incidents and allow supervisors to screen for both "professional and unprofessional officer language." There are around 30 police departments currently use Truleo, according to the company.

Christopher J. Schneider, a professor at Canada's Brandon University who studies the impact of emerging technology on social perceptions of police, is skeptical the AI tools will fix the problems in policing because the findings might be kept from the public just like many internal investigations. "Because it's confidential," he said, "the public are not going to know which officers are bad or have been disciplined or not been disciplined."

Palworld, a viral "Pokemon with guns" game, has become Microsoft's biggest third-party launch on Game Pass. According to developer Pocketpair, the game sold 12 million copies on Steam and seven million on Xbox since its January 19 launch. A million of the copies were sold in its first eight hours. Engadget reports: In addition to being the biggest third-party Game Pass launch ever, Palworld had the largest third-party day-one launch on Xbox Cloud Gaming (included with Game Pass Ultimate). The game's highest peak since launch was nearly three million daily active users on Xbox. Microsoft says it was the most-played game on Xbox platforms during that period.

Palworld uses Pokemon-esque characters and themes -- enough to catch the attention of Nintendo's lawyers. It has battles with monsters similar to those in the creature-collecting series, including the ability to capture them inside a sphere after winning. But Palworld also includes biting social commentary and incorporates themes you'd never see in Pokemon -- like labor exploitation. "Don't worry, there are no labor laws for Pals," a game FAQ reads. One of the title's trailers showed a player circling hard-at-work Pals with an assault rifle. "Creating a productive base like this is the secret to living a comfortable life in Palworld," the narration reads.

YouTube has announced it has surpassed 100 million YouTube Music and YouTube Premium subscribers globally. Variety reports: The 100 million figure includes uses who are on free trials, according to YouTube. The company didn't break down how many are on YouTube Music versus YouTube Premium, the subscription service for ad-free viewing, background listening, offline video downloads and full access to YouTube Music. In November 2022, the company said YouTube Music and YouTube Premium topped 80 million paying subscribers combined.

The announcement comes after Alphabet, in reporting fourth-quarter 2023 earnings, boasted that YouTube and Google subscription services generated more than $15 billion in revenue last year. That includes YouTube Premium and YouTube Music, as well as YouTube TV and Google One cloud storage.

An anonymous reader shares a report: Microsoft's adoption of Rust continues apace if a posting on the IT titan's careers website is anything to go by. Although headcount at Microsoft might currently be down -- by two percent compared to the previous year -- recruitment persists at the Windows giant. In this case, the company is forming a team of Rustaceans to tackle a platform move away from C#.

The job, a principal software architect for Microsoft 365, has responsibilities that include "guiding technical direction, design and implementation of Rust component libraries, SDKs, and re-implementation of existing global scale C# based services to Rust." According to the post, the job lurks within the Substrate App Platform group, part of the Microsoft 365 Core Platform organization. The Substrate does the heavy lifting behind the scenes for Microsoft's cloud services, making a rewrite into Rust quite a statement of intent. Microsoft said: "We are forming a new team focused on enabling the adoption of the Rust programming language as the foundation to modernizing global scale platform services, and beyond."

For the first time, Microsoft's Q2 earnings report includes the impact of the company's $68.7 billion Activision Blizzard acquisition. "While Microsoft isn't breaking out specific numbers, it says that its overall gaming revenue increased by 49 percent, 44 points of which came from the 'net impact' of the Activision deal," reports Engadget. From the report: Microsoft's More Personal Computing division, which includes Xbox, Surface and Windows, was up 19 percent ($16.9 billion) since last year. The company says the Activision deal accounted for 15 points of that increase. It's a huge change for a division that's been severely impacted by dwindling PC sales (which affects Windows licenses and Surfaces) and struggling Xbox consoles. PC device revenues were down 9 percent for the quarter, while Xbox hardware sales were up 3 percent. Xbox content and services revenue is also up 61 percent since last year, 55 points of which comes from Activision. Overall, Microsoft reported revenues of $62 billion, up from $52.7 billion a year earlier. Microsoft's cloud division posted revenue growth of 28%, with its intelligent-cloud revenue up 20% to $25.9 billion. Meanwhile, its productivity and business-processes segment generated $19.2 billion.

The U.S. government in recent months launched an operation to fight a pervasive Chinese hacking operation that successfully compromised thousands of internet-connected devices, Reuters reported Tuesday, citing two Western security officials and another person familiar with the matter. From the report: The Justice Department and Federal Bureau of Investigation sought and received legal authorization to remotely disable aspects of the Chinese hacking campaign, the sources told Reuters. The Biden administration has increasingly focused on hacking, not only for fear nation states may try to disrupt the U.S. election in November, but because ransomware wreaked havoc on Corporate America in 2023.

The hacking group at the center of recent activity, Volt Typhoon, has especially alarmed intelligence officials who say it is part of a larger effort to compromise Western critical infrastructure, including naval ports, internet service providers and utilities. While the Volt Typhoon campaign initially came to light in May 2023, the hackers expanded the scope of their operations late last year and changed some of their techniques, according to three people familiar with the matter. The widespread nature of the hacks led to a series of meetings between the White House and private technology industry, including several telecommunications and cloud commuting companies, where the U.S. government asked for assistance in tracking the activity.

An anonymous reader quotes a report from TechCrunch: Mercedes-Benz accidentally exposed a trove of internal data after leaving a private key online that gave "unrestricted access" to the company's source code, according to the security research firm that discovered it. Shubham Mittal, co-founder and chief technology officer of RedHunt Labs, alerted TechCrunch to the exposure and asked for help in disclosing to the car maker. The London-based cybersecurity company said it discovered a Mercedes employee's authentication token in a public GitHub repository during a routine internet scan in January. According to Mittal, this token -- an alternative to using a password for authenticating to GitHub -- could grant anyone full access to Mercedes's GitHub Enterprise Server, thus allowing the download of the company's private source code repositories.

"The GitHub token gave 'unrestricted' and 'unmonitored' access to the entire source code hosted at the internal GitHub Enterprise Server," Mittal explained in a report shared by TechCrunch. "The repositories include a large amount of intellectual property connection strings, cloud access keys, blueprints, design documents, [single sign-on] passwords, API Keys, and other critical internal information." Mittal provided TechCrunch with evidence that the exposed repositories contained Microsoft Azure and Amazon Web Services (AWS) keys, a Postgres database, and Mercedes source code. It's not known if any customer data was contained within the repositories. It's not known if anyone else besides Mittal discovered the exposed key, which was published in late-September 2023. A Mercedes spokesperson confirmed that the company "revoked the respective API token and removed the public repository immediately."

"We can confirm that internal source code was published on a public GitHub repository by human error. The security of our organization, products, and services is one of our top priorities. We will continue to analyze this case according to our normal processes. Depending on this, we implement remedial measures."

Long-time Slashdot reader theodp writes: IBM mainframes were the original onsite private cloud," begins retired software engineer Billy Newport in Is Cloud the New Mainframe? And while there were many things to like about the mainframe (including "crazy high availability numbers which today's cloud vendors can only dream of"), cost was not one of them. "As the application usage grows," Newport explains, "the bill grows and the control of the bill is largely in IBM's hands. You use more, you pay more [...] Unfortunately, while compute is elastic, budgets are not [...] Inevitably, customers try to migrate workloads from the mainframe to 'cheaper' platforms but these projects can be very expensive to do and they do fail more often than people realize."

"Today's Cloud kind of looks exactly the same as the mainframe scenario," Newport warns. "Companies have rushed to get on the cloud with the cool kids. I predict many companies will try to rush to reduce cloud expenditure and will find migrating onsite to be an expensive proposition if it's even possible.

"Companies that once serviced the boom in cryptocurrency mining are pivoting to take advantage of the latest data gold rush," reports the Guardian. Canadian company Hive Blockchain changed its name in July to Hive Digital Technologies and announced it was pivoting to AI. "Hive has been a pioneering force in the cryptocurrency mining sector since 2017. The adoption of a new name signals a significant strategic shift to harness the potential of GPU Cloud compute technology, a vital tool in the world of AI, machine learning and advanced data analysis, allowing us to expand our revenue channels with our Nvidia GPU fleet," the company said in its announcement at the time. The company's executive chairman, Frank Holmes, told Guardian Australia the transition required a lot of work. "Moving from mining Ethereum to hosting GPU cloud services involves buying powerful new servers for our GPUs, upgrading networking equipment and moving to higher tier data centres," he said.

"The only commonality is that GPUs are the workhorses in both cases. GPU cloud requires higher end supporting hardware and a more secure, faster data centre environment. There's a steep learning curve in the GPU cloud business, but our team is adapting well and learning fast."

For others, like Iris Energy, a datacentre company operating out of Canada and Texas, and co-founded by Australian Daniel Roberts, it has been the plan all along. Iris did not require any changes to the way the company operated when the AI boom came along, Roberts told Guardian Australia. "Our strategy really has been about bootstrapping the datacentre platform with bitcoin mining, and then just preserve optionality on the whole digital world. The distinction with us and crypto-miners is we're not really miners, we're datacentre people." The company still trumpets its bitcoin mining capability but in the most recent results Iris said it was well positioned for "power dense computing" with 100% renewable energy. Roberts said it wasn't an either-or situation between bitcoin mining and AI.

"I think when you look at bitcoin versus AI, the market will just reach equilibrium based on the market-based demands for each product," he said... Holmes said Hive also saw the two industries operating in parallel. "We love the bitcoin mining business, but its revenue is rather unpredictable. GPU cloud services should complement it well," he said.
Thanks to long-time Slashdot reader mspohr for sharing the article.

"Parched conditions have crippled a waterway that handles $270 billion a year in global trade," reports Bloomberg. "And there are no easy solutions.

"The Panama Canal Authority is weighing potential fixes that include an artificial lake to pump water into the canal and cloud seeding to boost rainfall, but both options would take years to implement, if they're even feasible. " With water levels languishing at six feet (1.8 meters) below normal, the canal authority capped the number of vessels that can cross. The limits imposed late last year were the strictest since 1989... Some shippers are paying millions of dollars to jump the growing queue, while others are taking longer, costlier routes around Africa or South America. The constraints have since eased slightly due to a rainier-than-expected November, but at 24 ships a day, the maximum is still well below the pre-drought daily capacity of about 38. As the dry season takes hold, the bottleneck is poised to worsen again...

The canal's travails reflect how climate change is altering global trade flows. Drought created chokepoints last year on the Mississippi River in the US and the Rhine in Europe. In the UK, rising sea levels are elevating the risk of flooding along the Thames. Melting ice is creating new shipping routes in the Arctic. Under normal circumstances, the Panama Canal handles about 3% of global maritime trade volumes and 46% of containers moving from Northeast Asia to the US East Coast...

In the long term, the primary solution to chronic water shortages will be to dam up the Indio River and then drill a tunnel through a mountain to pipe fresh water 8 kilometers (5 miles) into Lake Gatún, the canal's main reservoir. The project, along with additional conservation measures, will cost about $2 billion, Erick Córdoba, the manager of the water division at the canal authority estimates. He says it will take at least six years to dam up and fill the site. The US Army Corps of Engineers is conducting a feasibility study. The Indio River reservoir would increase vessel traffic by 11 to 15 a day, enough to keep Panama's top moneymaker working at capacity while guaranteeing fresh water for Panama City...

The country will need to dam even more rivers to guarantee water through the end of the century.

"Security experts expect many more companies to disclose that they've been hacked by Russian intelligence agents who stole emails from executives," reports the Washington Post, "following disclosures by Microsoft and Hewlett-Packard Enterprise in the past week." Microsoft said late Thursday that it had found more victims and was in the process of notifying them. A spokesperson declined to say how many. But three experts in and out of government said that the attack was deeper and broader than the disclosures to date reveal. Two said that more than 10 companies, and perhaps far more, are expected to come forward...

The Securities and Exchange Commission last year strengthened the rules that require companies to notify their stockholders of computer intrusions that could have a material impact on company results. That helped spur the recent disclosures.
A spokesperson for America's Department of Homeland Security said "at this time we are not aware of impacts to Microsoft customer environments or products," according to the article. (Although the Washington Post adds that "The Microsoft and HPE breaches are especially concerning because so many other companies and agencies rely on them for cloud services, including email.")

The attackers were potentially spying on Microsoft's senior leadership team "for weeks or months," reports the Verge, citing a newly-published analysis by Microsoft: Crucially, the non-production test tenant account that was breached didn't have two-factor authentication enabled. [A cyber-breaching group named Nobelium from Russia's foreign intelligence service] "tailored their password spray attacks to a limited number of accounts, using a low number of attempts to evade detection," says Microsoft. From this attack, the group "leveraged their initial access to identify and compromise a legacy test OAuth application that had elevated access to the Microsoft corporate environment...." This elevated access allowed the group to create more malicious OAuth applications and create accounts to access Microsoft's corporate environment and eventually its Office 365 Exchange Online service that provides access to email inboxes...

Hewlett Packard Enterprise (HPE) revealed earlier this week that the same group of hackers had previously gained access to its "cloud-based email environment." HPE didn't name the provider, but the company did reveal the incident was "likely related" to the "exfiltration of a limited number of [Microsoft] SharePoint files as early as May 2023."

Starting today Apple is opening up its App Store to allow game streaming apps and services. From a report: This means that services like Xbox Cloud Streaming and GeForce Now, which previously were only accessible on iOS via a web browser, will be able to offer full-featured apps. "Developers can now submit a single app with the capability to stream all of the games offered in their catalog," Apple wrote in a blog post. These changes apply "worldwide," according to the company.

In 2020, Apple appeared to have carved out a space for these cloud gaming services in the App Store. But that turned out not to be the case, as all games available through each service had to be submitted and reviewed as a standalone app. So the shift to allow one app with a large catalog of games marks a major change. As part of today's announcement, Apple said that "each experience made available in an app on the App Store will be required to adhere to all App Store Review Guidelines and its host app will need to maintain an age rating of the highest age-rated content included in the app." Apple also says that developers will now "be able to provide enhanced discovery opportunities for streaming games, mini-apps, mini-games, chatbots, and plug-ins that are found within their apps," and that "mini-apps, mini-games, chatbots, and plug-ins will be able to incorporate Apple's In-App Purchase system to offer their users paid digital content or services for the first time, such as a subscription for an individual chatbot."

The Wall Street Journal reports: Moves by Broadcom to shore up its $69 billion VMware acquisition, completed in November, include a streamlining of product bundles and new billing models — efforts in line with the chip giant's past acquisitions, but not necessarily welcomed by all of VMware's customers... Broadcom has also recently laid off at least hundreds of VMware workers, disclosures from the Worker Adjustment and Retraining Notification show....

VMware has approximately 330,000 customers, according to the company. Chief information officers say they are closely monitoring what comes next.

"Any CIO that's not taking stock of what they have and mentally considering alternatives and monitoring what else is out there is probably not doing their job," said Jay Ferro, executive vice president and chief information, technology and product officer at clinical research data-management company Clario. All these changes, plus past remarks by Broadcom that its go-to-market strategy is to focus completely on the needs and priorities of its top 600 customers, has left some CIOs rethinking the relationship. Price increases and degrading levels of support are among their biggest concerns. "I'm not one of their top, probably 600 customers, so they've been very clear to me where I fit in that pecking order," said Todd Florence, CIO of trucking company Estes Express Lines. Florence said he's started looking into alternatives. "It certainly doesn't make you feel good, like you're going to get lots of support going forward...."

Goya Foods CIO Suvajit Basu said he is thinking about how to reduce the food company's reliance on VMware as the sole and longtime dominant provider of virtualization for the data center. "They're going to increase their prices or change their licensing so the customer pays more," he said. "And I think this is starting to hit us right now...." Forrester estimates that in 2024, 20% of VMware customers will begin the process of exiting VMware in favor of alternatives.
On the other hand, a group VP at market researcher IDC tells the Journal that on the upside, now VMware and Broadcom will have to engage more actively with customers on the value of new produces included in their bundles...

It's "a free and open-source, software-defined storage platform," according to Wikipedia, providing object storage, block storage, and file storage "built on a common distributed cluster foundation". The charter advisory board for Ceph included people from Canonical, CERN, Cisco, Fujitsu, Intel, Red Hat, SanDisk, and SUSE.

And Nite_Hawk (Slashdot reader #1,304) is one of its core engineers — a former Red Hat principal software engineer named Mark Nelson. (He's now leading R&D for a small cloud systems company called Clyso that provides Ceph consulting.) And he's returned to Slashdot to share a blog post describing "a journey to 1 TiB/s". This gnarly tale-from-Production starts while assisting Clyso with "a fairly hip and cutting edge company that wanted to transition their HDD-backed Ceph cluster to a 10 petabyte NVMe deployment" using object-based storage devices [or OSDs]...) I can't believe they figured it out first. That was the thought going through my head back in mid-December after several weeks of 12-hour days debugging why this cluster was slow... Half-forgotten superstitions from the 90s about appeasing SCSI gods flitted through my consciousness...

Ultimately they decided to go with a Dell architecture we designed, which quoted at roughly 13% cheaper than the original configuration despite having several key advantages. The new configuration has less memory per OSD (still comfortably 12GiB each), but faster memory throughput. It also provides more aggregate CPU resources, significantly more aggregate network throughput, a simpler single-socket configuration, and utilizes the newest generation of AMD processors and DDR5 RAM. By employing smaller nodes, we halved the impact of a node failure on cluster recovery....

The initial single-OSD test looked fantastic for large reads and writes and showed nearly the same throughput we saw when running FIO tests directly against the drives. As soon as we ran the 8-OSD test, however, we observed a performance drop. Subsequent single-OSD tests continued to perform poorly until several hours later when they recovered. So long as a multi-OSD test was not introduced, performance remained high. Confusingly, we were unable to invoke the same behavior when running FIO tests directly against the drives. Just as confusing, we saw that during the 8 OSD test, a single OSD would use significantly more CPU than the others. A wallclock profile of the OSD under load showed significant time spent in io_submit, which is what we typically see when the kernel starts blocking because a drive's queue becomes full...

For over a week, we looked at everything from bios settings, NVMe multipath, low-level NVMe debugging, changing kernel/Ubuntu versions, and checking every single kernel, OS, and Ceph setting we could think of. None these things fully resolved the issue. We even performed blktrace and iowatcher analysis during "good" and "bad" single OSD tests, and could directly observe the slow IO completion behavior. At this point, we started getting the hardware vendors involved. Ultimately it turned out to be unnecessary. There was one minor, and two major fixes that got things back on track.
It's a long blog post, but here's where it ends up:

• Fix One: "Ceph is incredibly sensitive to latency introduced by CPU c-state transitions. A quick check of the bios on these nodes showed that they weren't running in maximum performance mode which disables c-states."

• Fix Two: [A very clever engineer working for the customer] "ran a perf profile during a bad run and made a very astute discovery: A huge amount of time is spent in the kernel contending on a spin lock while updating the IOMMU mappings. He disabled IOMMU in the kernel and immediately saw a huge increase in performance during the 8-node tests." In a comment below, Nelson adds that "We've never seen the IOMMU issue before with Ceph... I'm hoping we can work with the vendors to understand better what's going on and get it fixed without having to completely disable IOMMU."

• Fix Three: "We were not, in fact, building RocksDB with the correct compile flags... It turns out that Canonical fixed this for their own builds as did Gentoo after seeing the note I wrote in do_cmake.sh over 6 years ago... With the issue understood, we built custom 17.2.7 packages with a fix in place. Compaction time dropped by around 3X and 4K random write performance doubled."

The story has a happy ending, with performance testing eventually showing data being read at 635 GiB/s — and a colleague daring them to attempt 1 TiB/s. They built a new testing configuration targeting 63 nodes — achieving 950GiB/s — then tried some more performance optimizations...